The Quick Guide To Making Your Own Dataset With Python

Collect and store data from your users using the Google Sheets API and Streamlit

Photo by Marcus Loke on Unsplash

Have you ever wondered how hard would it be to collect data from your users and thus try to build your own dataset?

In this article, I go over a simple data collection process that you can apply for your own use case and easily collect and store data, all under an hour.

We use **Streamlit for building the webpage to host our data collection user interface and then use the [Google Sheets API](developers.google.com/sheets/api/reference/.. along with one cool Python package to store the user-entered data.

Let’s get started 👇

First step — Building the UI

Streamlit is a neat Python tool to easily spin-up interactive, aesthetically decent web applications. You can install with one line:

pip install streamlit

Now, we can start making a script to include some text and other data collection widgets in our webpage!

If you are new to Streamlit, I have a simple, starter guide that you can follow to get yourself up to speed. Here’s the article. You can come back to this article after you’ve read that one, or you can also follow along here, as it’s not going to be too advanced.

After making a new file app.py, let’s start with including a title for our page:

st.title(‘My Wholesome Data Capture Project’)

Then, you can include some description of what you’re trying to do for your user:

st.write("My app wants to collect data from you about...")

Next, you can start collecting user inputs using simple widgets, such as:

st.subheader('Add a new item') 
name = st.text_input('Enter the item name') # to collect text input

You have a variety of widgets to choose from such as select-boxes, text-area, sliders, file chooser, etc, and I’ve touched upon most of them in my article here.

Once you’ve built them satisfactorily, go ahead and start your streamlit server:

streamlit run app.py

Now, you should be able to see the output in your browser by navigating to localhost:8000!

Connecting the Google Sheets API to our app

There are two steps to connecting the app to Google Sheets in order to collect data.

1. Enable the API from your Google Cloud console: Go to this link and choose enabling the Google Sheets API.

The window should appear like this:

Enabling the sheets API

1. Click on Enable. If you haven’t created a new project in the Google cloud console yet, go ahead and do that now.

2. Next, click on Credentials and then Create an API Key.

3. Press Manage service accounts above Service Accounts.

4. Press on the ⋮ button near recently created service account and select Manage keys and then click on ADD KEY → Create new key.

5. Select JSON key type and click Create.

creating a JSON key for your service account

You will now get a json document to download which will contain all your service account credentials to connect to your Streamlit Python app to Google Sheets app.

Save it in your project folder for now.

Final Step — Installing gspread

This cool python library enables easier integration of Google Sheets in our Python applications. Install it simply:

pip install gspread

You can open up a new Google Sheet in the browser and enable link sharing. This is important as it will allow the sheets API to write data into the spreadsheet.

Now, we can connect our spreadsheet with our streamlit application.

Into app.py, write a new function called ‘**save_into_csv**’!

import gspread

def save_into_csv(name, author, link, pair, summ):    
    gc = gspread.service_account(filename='credentials.json')     
    sh = gc.open_by_url('your google sheet url')    
    worksheet = sh.get_worksheet(0)

Here, make sure your credentials.json from Sheets API is in the same directory as your app.py file.

Now, you can begin writing the user captured input data into your spreadsheet:

worksheet.append_row([your user input items])

Make sure to set column headers inside your spreadsheet first, if you want, before using the app! That way, all the captured user data will get saved into corresponding columns under your defined column headers.

As a bonus, it’s probably a good idea to display the submitted information to the user too. You can just type out a message with Streamlit like:

st.write('Submitted to database!')

Concluding…

And that is all it takes! You are ready to collect your user data and make your own dataset! :D

I hope this tutorial was useful, and you now feel able to delve deeper into some more nuances of the Sheets API, as well as the gspread library. There’s much to explore, if you want to, and after this beginner project, you should be ready to do it!

Loved my article? Become a Medium member to continue learning without limits. I’ll receive a portion of your membership fee if you use that link, with no extra cost to you. It will help me support my writing endeavour :)

If you like reading from me, consider following me as well.

There’s also the awesome collection of data science resources that I’ve compiled from my articles in this repository. Go check it out!

A couple of my articles you might also want to read: The Reusable Python Logging Template For All Your Data Science Apps towardsdatascience.com The Nice Way To Use Docker With VSCode towardsdatascience.com