Count how many different items appear in any nested embedded List Python

Tutorial

Life is a like an onion, you peel it off one layer at time, and sometimes you weep Carl Sandberg

I suppose the same could be said of extracting values from nested JSON structures. Even the most skilled programmer can be brought to tears when working with a JSON object that consists of a mix of deeply nested data structures. The process of extracting the values can feel messy and disorganized at best. The more data there is, the bigger the mess.

In this tutorial, Ill walk you through a step-by-step method to extract the values you need from any JSON. A word of warning: this tutorial is not meant for newbies to JSON, lists or dictionaries. If youve never heard of a list index or a dictionary key-value pair, I would suggest reviewing one of the many great tutorials available on the web or YouTube. Once you feel more comfortable with the subject, come back to continue learning and growing.

Housekeeping

JSON vs. Lists vs. Dictionaries

First things first, when it comes to the terms JSON, list and dictionary, we have to do some important housekeeping. JSON, or JavaScript Object Notation, is a broader format used to encompass dictionary and list structures as shown in the image below.

The technical documentation says a JSON object is built on two structures: a list of key-value pairs and an ordered list of values. In Python Programming, key-value pairs are dictionary objects and ordered list are list objects. In practice, the starting point for the extraction of nested data starts with either a dictionary or list data structure. When extracting Nested Data the question[s] should be: Is the data nested in a Dictionary or List data structure? What is the combination of data structures used? Is the first data structure used a dictionary or a list?

It has long been an axiom of mine that the little things are infinitely the most important. Sir Arthur Conan Doyle,

If it seems like Im making a big deal about the terminology, it is because I am. When it comes to extracting nested data the details matter. Data structures change the deeper the data is nested in the JSON structure and knowing those distinctions are important. The initial data structure may be a list but then change to a dictionary as the data is extracted. The key to extracting data from a JSON object is recognizing the mix of data structures used to store the data. If you struggle to recognize the data structure in a JSON object, its likely that youll struggle to extract the values you want. In most cases, this results in applying the wrong extraction technique.

The table below is a brief refresher on the techniques used to extract data from a JSON structure.

One final note before starting our example. In Python Programming, the term data structure is rarely used when describing lists and dictionary. The commonly used term is data type. I use the terms data type and data structure interchangeably throughout this tutorial. I use the term data structure because it conveys the idea that the data structures are the fundamental building blocks of the JSON object. The usage of the term data type in Python is not of less importance however it does not convey the same meaning as a key to understanding nested data extraction.

Real World Data

Let's Get Started

One of the best ways to learn is by working through real data with a mix of list and dictionary data structures. In this tutorial, well use real data from the REST Countries API. This API returns about 250 records with a mix of dictionaries, lists and other data types. Our objective is to extract the 'AFN value from the dictionary key-value pair 'code':'AFN' as shown in the image below. The 'AFN' is nested in two list structures and one dictionary structure.

The Sample Code

Clicking this link will allow you to access the sample code in the following examples. The filenames are single_json.py and multiple_json.py.

Extracting Single Items

In this example, well start by extracting data using a combination of list and dictionary extraction techniques as shown in the preceding table. In the Python code below, I start by providing the logic to import the data, the solution and then the workflow to derive the solution. I recommend following all the steps as shown below. The workflow steps are explained below the Python code.

Python Code:

Workflow Steps:

• Step 1: import requests: this line imports the Requests HTTP library for Python. It is the library we use to connect to a Restful API. If you havent already installed it, you can install it from the command prompt or virtual environment using the pip install requests command.
• Step 2: url = '//restcountries.eu/rest/v2/all' this line stores the web address for the REST API. The address is stored in the url variable.
• Step 3: response = requests.get[url]: this method is used to connect to the Restful API, //restcountries.eu/rest/v2/all, to extract the data. The data returned is stored in the response variable. In technical terms, this is referred to as the response object.
• Step 4: storage = response.json[]returns a JSON object of the result [if the result was written in JSON format, if not it raises an error]. Think of the .json[]as a storage format used to exchange the data. In this instance, we store the content in the storage variable.
• Step 5: print[type[storage]]: this returns the Python data type used to store the data. In this instance, the data type returned will be a list [ ]. Looking back the table I provided earlier, the data can be extracted using the list index [0,]. You should always use the type[]function to determine the data type. If you know the data type, you know the correct extraction technique to use.
• Step 6: print[len[storage]]: this provides the number of items in the list. Each number represents the index of an item in the list and the index can be used to extract the value.
• Step 7: print[storage[0]]: the 0 represents the first item in the list AND is used to extract the item from the list. Once the item is extracted, a new data type is now exposed. The type[]function in step 8 is used to show the data type.
• Step 8: print[type[storage[0]]]: the new data type will be