import requests
from os import makedirs
from os.path import join
from shutil import unpack_archive
from glob import glob
SOURCE_URL = 'https://www.ssa.gov/oact/babynames/names.zip'
DATA_DIR = 'tempdata'
DATA_ZIP_PATH = join(DATA_DIR, 'names.zip')
# make the directory
makedirs(DATA_DIR, exist_ok=True)

print("Downloading", SOURCE_URL)
resp = requests.get(SOURCE_URL)
# save it to disk
# we use 'wb' because these are BYTES
with open(DATA_ZIP_PATH, 'wb') as f:
    # we use resp.content because it is BYTES
    f.write(resp.content)

# now let's unzip it into tempdata/
unpack_archive(DATA_ZIP_PATH, extract_dir=DATA_DIR)

# get all the filenames
babynamefilenames = glob(join(DATA_DIR, '*.txt'))

You can glob for filenames like this:

from os.path import join, basename
from glob import glob
DATA_DIR = 'tempdata'
alltxtfiles_path = join(DATA_DIR, '*.txt')
alltxtfiles_names = glob(alltxtfiles_path)

Filtering to include only the data files that are after 1950 is a little tricky.

But we can use the filenames themselves to our advantage. Remember, they’re just strings.

And they look like this:

• tempdata/yob1949.txt
• tempdata/yob1950.txt
• tempdata/yob1951.txt

If you know how to use regular expressions, go to town. Otherwise, consider this step by step deconstruction of the filename:

from os.path import basename

a_filename = 'tempdata/yob1951.txt'
bname = basename(a_filename)
# bname is now: yob1951.txt
# the first digit is at the index position of 3
# and we want the next 4 characters, so....
year = bname[3:7]
# bingo: year is the string "1951"

Basically, you want to filter those files you previously globbed (i.e. alltxtfiles_names) by testing if the year component of the file name is greater-than-or-equal-to 1950. If you don’t convert it to an integer, the comparison is with the string “1950”:

myfilenames = []
for fname in alltxtfiles_names:
  bname = basename(fname) # e.g. "yob1980.txt"
  year = bname[3:7]       # e.g.    "1980"
  if year >= "1950":
      myfilenames.append(fname)

Now, do the counting of baby names using the filtered list in myfilenames

Here’s one approach:

totalsdict = {'M': 0, 'F': 0}

for fname in myfilenames:
    babyfile = open(fname, "r")
    for line in babyfile:
        name, gender, babies = line.split(',')
        # need to convert babies to a number before adding
        totalsdict[gender] += int(babies)

# Now, totalsdict contains two keys, 'M' and 'F', which both point
# to very large integers

print("F:", str(totalsdict['F']).rjust(6),
      "M:", str(totalsdict['M']).rjust(6))


f_to_m_ratio = round(100 * totalsdict['F'] / totalsdict['M'])
print("F/M baby ratio:", f_to_m_ratio)

The same approach as b.py, except instead of tallying total baby counts, you are simply counting all unique names over the years.

How to keep a collection of unique items? Try using the set collection. The set object has an add() method to add one object a time. If the object already exists, it won’t expand the set’s collection.

To get the number of items in a set, just use the len() function:

tally = {'M': set(), 'F': set()}
tally['F'].add("Lisa")
tally['F'].add("Lisa")
tally['F'].add("Lisa")
print(len(tally['F']))
# 1

Here’s 60% of the solution (well, my solution):

from os.path import join, basename
from glob import glob
DATA_DIR = 'tempdata'

# unlike b.py, this simply keeps a count of "names", not total babies
tally = {'M': set(), 'F': set()}

for fname in glob(join(DATA_DIR, '*.txt')):
    # doing the filtering if filenames in one loop
    year = basename(fname)[3:7]
    if year >= "1950":
        for line in open(fname, 'r'):
            name, gender, babies = line.split(',')
            tally[gender].add(name)

The tally object, i.e. the dictionary of sets, can be used just as it was in c.py:

tally = {'M': set(), 'F': set()}

But just like the print statements, the object must be initialized with each iteration of the for-loop, i.e. with every new year file.

There’s countless ways to do this. The variations in which are not particularly important.

Below is my complete answer with rambling comments. If you follow it strictly, please at least don’t copy the comments.

from os.path import join, basename
# we're only dealing with one file, i.e. yob2014.txt
# but it's worth storing its "year" value in a variable just to abstract things
YEAR = 2014
DATA_DIR = 'tempdata'
thefilename = join(DATA_DIR, 'yob' + str(YEAR) + '.txt')

names_dict = {}
thefile =  open(thefilename, 'r')
for line in thefile:
    name, gender, count = line.split(',')
    # Every name can show up twice for a year, as M or F
    # but some names only show up for either M or F
    # that means we need to initialize a new value
    # for a given name in names_dict if it doesn't already exist
    if not names_dict.get(name):
        # i.e. names_dict does not yet have `name` as a valid key
        # so we make it a valid key by initializing it and pointing it to
        # a dictionary that we can add values to
        names_dict[name] = {'M': 0, 'F': 0}

    # Now that names_dict[name] is itself a dictionary, {'M': 0, 'F': 0}
    # we can safely add the `count` variable to it
    #  e.g. names_dict['Jennifer']['F'] = int("24222")
    names_dict[name][gender] += int(count)
# at this point, when the for loop is done
# we're done reading from the file, so we can close it
thefile.close()

# names_dict now contains a dict of dicts:
# {
#    'Jennifer': {'F': 24222, 'M': 32},
#    'Amanda':   {'F': 10000, 'M': 0 },
#    'John':     {'F': 12,    'M': 12000}
# }


############################################################
# Now it's time to print things...
# The year 20YY has XXXXX unique names for ZZZZ total babies

# which means we need to get the total number of baby names,
# which is simply a len() call on the keys of names_dict
total_namecount = len(names_dict.keys())

# and then we need to get the total baby count...here's one straightforward
# way to do it:
total_babycount = 0
for v in names_dict.values():
    totes = v['M'] + v['F'] # count up males and females
    # and add it to the total_babycount
    total_babycount += totes

# or, you could've done this:
# ...sum(v['F'] + v['M'] for v in names_dict.values())

print("Total:", total_namecount, 'unique names for', total_babycount, 'babies')

# now we do the same thing, except for just boys and their names
ncount = 0
bcount = 0
for v in names_dict.values():
    # don't count it as a boy name if no babies were actually given the name
    if v['M'] > 0:
        bcount += v['M']
        ncount += 1
print("    M:", ncount, "unique names for", bcount, "babies")


# now we do the same thing, except for just girls and their names
ncount = 0
bcount = 0
for v in names_dict.values():
    # don't count it as a girl name if no babies were actually given the name
    if v['F'] > 0:
        bcount += v['F']
        ncount += 1
print("    F:", ncount, "unique names for", bcount, "babies")


# or if you wanted to be thrifty and not repeat yourself, do a for loop:
# for gender in ['M', 'F']:
#     ncount = 0
#     bcount = 0
#     for v in names_dict.values():
#         # don't count it as a girl name if no babies were actually given the name
#         if v[gender] > 0:
#             bcount += v[gender]
#             ncount += 1
#     print("    %s:" % gender, ncount, "unique names for", bcount, "babies")

Here’s one way to print the number of babies per name ratio at the end, the total followed by the breakdown for both genders (this comes at the end, so come up with your own variables):

print("Total:", round(total_babycount / total_namecount), 'babies per name')
# for boys and girls separately
for gd in ['M', 'F']:
    babyct = 0
    namect = 0
    for v in names_dict.values():
        if v[gd] > 0:
            babyct += v[gd]
            namect += 1
    print("    %s:" % gd, round(babyct / namect), 'babies per name')

Data-wrangling, which is often what people think of when they think of data-cleaning, is one of the most difficult programmatic tasks, in the sense that naming things and cache invalidation is difficult in computer science. There’s not just one way to do it, and there’s not one clear, absolutely superior goal.

To make things easy, I’ve set a relatively simple and straightforward goal. It certainly has its flaws, which may become evident when trying to use it in real-world analysis. But creating it is relatively straightforward, at least in my muddled mind.

That said, to reduce confusion, I’ll provide my answer, which is a bit verbose, but hopefully makes it clear that this is just the same kind of data manipulation and handling we’ve done before in Python.

The start

Nothing fancy – for this exercise, we’ll be working only with yob2014.txt, with the knowledge that if we can deal with one file, we can deal with every file as we please:

from os.path import join, basename
import csv
DATA_DIR = 'tempdata'
YEAR = 2014
thefilename = join(DATA_DIR, 'yob' + str(YEAR) + '.txt')

Setting up the wrangled file

Let’s create a constant for the new file we’ll be making. In fact, let’s create another constant that stores the list of column names this new file will have:

WRANGLED_HEADERS = ['year', 'name', 'gender' , 'ratio' , 'females', 'males', 'total']
WRANGLED_DATA_FILENAME = join(DATA_DIR, 'wrangled2014.csv')

Gathering up the name data

This step is exactly the same as it is for every previous exercise: for every name, collect the number of babies by gender.

namesdict = {}
with open(thefilename, 'r') as thefile:
    for line in thefile:
        name, gender, count = line.split(',')
        if not namesdict.get(name): # need to initialize a new dict for the name
            namesdict[name] = {'M': 0, 'F': 0}
        namesdict[name][gender] += int(count)

Let’s make a list

The object namesdict has been perfectly servicable as a dictionary of dicts; in fact, we could probably continue using it without too much trouble. If you’ve been following along in Interactive Python, you can inspect it and see something like this:

{
 'Taytem': {'F': 9, 'M': 0},
 'Favour': {'F': 19, 'M': 0},
 'Yitzchok': {'F': 0, 'M': 119},
 'Daymon': {'F': 0, 'M': 25}
}

However, what we eventually need is a collection of dictionaries with different attribute names and more attributes.

So I’ve opted to just create a new list and then append it full of dictionary objects that have all the headers and values we need.

It starts like this:

my_awesome_list = []

Each dictionary we want to add to my_awesome_list will contain values derived from each key-value pair in namesdict.

So, basically a for-loop:

for name, counts in namesdict.items():
    xdict = {}
    xdict['year'] = YEAR # i.e. 2014
    xdict['name'] = name
    xdict['females'] = counts['F']
    xdict['males'] = counts['M']
    xdict['total'] = xdict['males'] + xdict['females']
    # the "likely" gender is determined by comparing females vs males numbers
    if xdict['females'] >= xdict['males']:
        xdict['gender'] = 'F'
        xdict['ratio'] = round(100 * xdict['females'] / xdict['total'])
    else:
        xdict['gender'] = 'M'
        xdict['ratio'] = round(100 * xdict['males'] / xdict['total'])

    # finally, add our new dict, xdict, to my_awesome_list
    my_awesome_list.append(xdict)

There are some questions worth asking. For example, we used to track the count of female and male babies with the F and M keys, like this:

{'Daniel': {'F': 100, 'M': 9000}}

Why the change to females and males?

{'name': 'Daniel', 'females': 100, 'males': 9000}}

It’s a matter of opinion. But remember that we’re creating a new data file. And when a user comes upon it, what’s going to make more sense when looking at the headers:

  name,F,M
  Daniel,100,9000

Or:

  name,females,males
  Daniel,100,9000

It requires a little more work in organizing the data, but part of data-wrangling is producing a more useful public face that may not have been necessary when initially working with the data.

Creating a new file and using csv.DictWriter()

This is basically just a pattern you memorize: when serializing a list of dictionaries as a flat CSV file, you use csv.DictWriter, that’s all. That said, it took me quite a few tries to memorize it. Hopefully it’s clear why we created easy to remember constants for WRANGLED_DATA_FILENAME and WRANGLED_HEADERS:

# let's create the new file to write to
wfile = open(WRANGLED_DATA_FILENAME, 'w')
# turn it into a DictWriter object, and tell it what the fieldnames are
wcsv = csv.DictWriter(wfile, fieldnames=WRANGLED_HEADERS)
# write the headers row
wcsv.writeheader()

Sorting the data before writing it

Oh but we can’t write the actual data rows just yet…As more pain-in-the-butt requirement, we’re required to sort the rows in order of the total column (in descending order), then by the name column. Do it how you like, this is how I did it:

def xfoo(xdict):
    # and return a tuple of negative total, and normal name
    return (-xdict['total'], xdict['name'])

my_final_list = sorted(my_awesome_list, key=xfoo)
for row in my_final_list:
    wcsv.writerow(row)
# the end...close the file
wfile.close()

Write the first five lines of text

Just to make sure that we’ve produced the file we want, this exercise asks us to re-open the text file – but not to parse it into data – but just to print the first five lines as plain text. This works:

finalfile = open(WRANGLED_DATA_FILENAME, 'r')
thestupidlines = finalfile.readlines()[0:5]
for line in thestupidlines:
    # remember each text line has a newline character
    # that we don't want to print out for aesthetic reasons
    print(line.strip())

And that should get you to the desired output.

All of the code from h.py that is used to deserialize the data into a list of dictionaries can be reused here.

You have one additional data processing step: filtering that list to include only names that have 100 or more total babies. You should be able to figure this out:

    bigdatarows = []
    for row in datarows:
        if SOMETHINGSOMETHING
              bigdatarows.append(row)

The number of “popular” names in 2014 is simply len(bigdatarows)

As for what you need to print out, the process is about the same as it was in the previous exercise. But you start off with a for-loop:

    print("Popular names with a gender ratio bias of less than or equal to:")
    for genderratio in (60, 70, 80, 90, 99):

This is barely an exercise. It’s meant to serve as another example in programming of how once you get something working once – there’s no reason why you can apply the same operation across many values or files. So why restrict ourselves to wrangling just the 2014 file when we can literally do it for every other baby name data file?

The beginning of this script looks very much the same as it did in g.py, though note we’re saving to a different file name: /tempdata/wrangledbabynames.csv:

from os.path import join, basename
import csv
DATA_DIR = 'tempdata'
# as before, we create new headers for our wrangled file
# though we leave out year because we probably don't care at for our ultimate needs
WRANGLED_HEADERS = ['name', 'gender' , 'ratio' , 'females', 'males', 'total']
WRANGLED_DATA_FILENAME = join(DATA_DIR, 'wrangledbabynames.csv')

Since we’re reading from multiple years`, we need to create a list of numbers, starting from 1950 and ending at 2014:

This is how to produce the list of numbers using a range:

for year in range(1950, 2014, 10):
    print(year)

1950
1960
1970
1980
1990
2000
2010

Note that it stops before 2014, so we just have to add that manually to the list:

START_YEAR = 1950
END_YEAR = 2014
# lets just get a list of all decades, between 1950 and 2014:
years = list(range(START_YEAR, END_YEAR, 10))
# and let's tack on the END_YEAR manually:
years.append(END_YEAR)

Also note that the interval and number of years is arbitrary. You could just as easily aggregate every file since 1880 if you wished.

After that, things are largely the same. We loop through every year, but the namesdict collection stays “above” it all because it is counting up names for every file:

namesdict = {}
for year in years:
    # get the file for this particular year
    filename = join(DATA_DIR, 'yob' + str(year) + '.txt')
    print("Parsing", filename)
    # fill in the rest yourself

Similarly, we wait till that year-loop is finished before creating my_awesome_list, which contains the same stuff as namesdict.

my_awesome_list = []
# just the same as it was for g.py, except no "year"
for name, babiescount in namesdict.items():
    xdict = {'name': name, 'females': babiescount['F'], 'males': babiescount['M']}

In fact, everything here on out should be pretty much the same as g.py.

Remember, you’re just doing what you did to the 2014 file, except to a bunch of files. That includes aggregating it into a single list at the end.