Skip to main content

Google Colab to Databricks

 

Google Colab to Databricks

Google Colab to Databricks


Hello Everyone,


Recently I started using a new tool for Data Analytics (Databricks Community Edition) and this post shares my experience in moving from Google Colab to Databricks.


What is Databricks Community Edition notebook?

It is a powerful platform for collaboration among data analysts, data scientists, and data engineers. You can think of it as a cloud-based Data Analytics Platform which gives you a chance to tap into Spark and other open-source tools.

 

First impressions on moving from Google Colab to Databricks

At first, it seems similar to Colab, the same notebook environment. But when you start working with it, you’ll notice the differences. The first major difference I noticed was the filesystem. Also, many other notebook features like Spark, SQL, and SQLAnalytics can be accessed for learning at no cost.

Another major difference is that the databricks has two filesystems, one local and another on AWS. If you want to upload any file to the environment, it gets uploaded on AWS and then it has to be copied to the local system using a set of commands (“%fs”, ” file: ”,  “dbfs: ”, ”cp”). All the library installation files are stored locally and not on AWS.

 

Why did I start using it?

Most of the time my first preference for Data Exploration or ETL is Google Colab. But, recently I started working full-time and this was the tool they preferred for data exploration. Personally, the start was a bit bumpy, but eventually, I got used to the tool. After a month of experience, I can say that the environment has more to it than I knew.

 

Initial challenges

Given the obvious differences in accessing external files, processing the uploaded files in the notebook was a bit of a challenge. I had to look around for the commands to transfer the uploaded files and also had to get a grasp of the basic workings to complete many tasks quickly.

Moreover, another observation was that sometimes the cells either gave inconsistent output or either they won’t run. In most of the cases, I had to log out and then log in to get the work done.

Also, sometimes the runtime environment would get detached from the notebook. So, I had to reattach the environment and wait a couple of seconds for it to be ready.

But, apart from these issues, the overall experience is good and amazing.

 

Last thoughts

Lastly, I think a comparison between Google Colab and Databricks Community Edition tools would not be fair. Because, one is a cloud data analytics platform with SQL and Spark, while the other is a tool used for solely leveraging the hardware power for completing your task.

 

My opinion

In my opinion, if you are learning and practicing Machine Learning or Deep Learning, Colab is a good tool for beginners. But, if you are a professional trying out solutions that involve Big Data, then Databricks would be a better choice.

 

Further readings

 

Sources

  1. https://docs.databricks.com/notebooks/index.html
  2. https://databricks.com/product/faq/community-edition

 

Labels:
Location: Toronto, ON, Canada

Comments

Post a Comment

Popular posts from this blog

PyMuPDF vs PDFMiner

 As a developer , I was tasked to extract specific data from a PDF. Upon analysing it further, certain patterns were found based on keywords in the document. Since I was using Python language for the task I found 2 tools quite useful which are PyMuPDF and PDFMiner. These tools can then be used to extract the text from a page on which regular expression can be applied to further extract relevant data.     Next, we are going to take a deeper look into these tools, specifically focusing on the pros and cons of each.     PyMuPDF   Docs , PIP package Pros Simple and understandable API Extensive tools to work with text, images, and graphics Available as a PIP package (pip install PyMuPDF) Better support for a range of symbols comparer to PyPDF2   Cons Parsed text is not in sequence Dependency on other package-Fitz Text sequence information lost during extraction     PDFMiner   Docs ,  PIP package ...
Post a Comment
Read more

Finding difference between 2 files in Python

In this post, we will take a look at how to compare two files using Python.   I was tasked to compare 2 files and then list the differences between them using Python. Initially, I started with filecmp module, but even with the function parameter ‘ shallow’ set to false, the Boolean result was not enough. Sure, it can act as an indicator to take some action, but it will not list the differences.   I was looking for something more visual, something like color coding and not like the git diff output, which is not very user-friendly. But, another Python internal module, difflib helped me to get the job done.   Inside Difflib, HtmlDiff is what I was looking for. The differences were highlighted with 3 different colors and also the line numbers were indicated in a table to locate the differences. The results are quite self-explanatory and it is easier to explain the differences to other people. Code for generating the above difference table: Note: File1...
Post a Comment
Read more

Adding existing Anaconda environment to Jupyter notebook

In this post we are going to take a look at adding Anaconda environment to Jupyter notebook. Recently, I was working on a CSV file and wanted to work with Pandas package for tabular data manipulation using Python. The problem was even if I install Pandas package, I would have to install other Data Science package as needed. But, the Anaconda environment was already setup on my laptop, which I want to reuse.   Today, we will look into how to reuse the Anaconda environment within the Jupyter Notebook.   There are 4 basic steps to be followed for adding the environment: 1. Create a conda environment Go to Conda command prompt(Run in Admin mode) Run the following command: conda create –-name newenv O/P:   What if there is an existing conda environment? Go to Conda command prompt(No need for Admin mode) Run the following command: conda env list O/P: Since there was only one environment, only one entry was displayed. ‘*’ indicates the cur...
Post a Comment
Read more