On using less popular R packages and data validation

Note: This post is geared towards data practitioners. If you don't fall into this group, feel free to ignore.

I have a saying about R (probably in Python, I know its in R) -- There's a package for everything. The beauty of open source languages is that anyone can write their own packages [libraries] and publish them on Github for anyone to download. The vast majority of useful R packages are collected and hosted in the CRAN package repository, which currently features over 18,000 packages.

When I say that there's a package for everything, I really mean it. There are the "standard" packages to augment base R, like the Tidyverse to help clean and structure data. But there are also more niche packages that can help you do useful things like import SAS files. There are also totally random packages, like the awesome Brooke Watson's package that can make your computer output Rapper Adlibs when your script is finished running (I think my favorite is Waka's, but the lack of Kanye in the package is borderline criminal).

This luxury comes with a price, however. Like our Middle School teachers warned us about Wikipedia,

Comments

Why isn't Robinhood letting me trade? (hint: there's probably not a conspiracy against you)

Today's been a big day in the stock market . Lots of people have lost a lot of money, and a lot of people are understandably really upset . Here's a quick breakdown of what's happened so far A subreddit called /r/wallstreetbets (visit at your own peril), which has exploded in popularity recently and has over 5 million subscribers (and counting) got really excited about three stocks: GME (Gamestop), AMC (the movie theater place), and BB (Blackberry). Gamestop was the main stock. Yes, I know all three companies are doing terribly in the real world. I won't go into why they got excited about the stocks here. They convinced a lot of other people to buy the stocks and they did well. Really well. Take a look at their Yahoo Finance pages and look at their 1 month price charts (then ignore the past two days). GME , BB , AMC Everyone got in on it, and I mean it. When a lot of people buy a single stock, the price rises. It turns out, this was hurting a lot of Hedge Funds and I

Determining NFL Quarterback Archetypes (with stats!)

We're obsessed with grouping things together. We self-select each other into groups based on which political candidate we support, which sports team we root for, and which arbitrary country we're born in. People also spend hours on the internet arguing over "tiers", or groupings, of their favorite athletes and sports teams. For example, which NBA players are "elite" vs. "great" vs. just "good"? Did Carmelo Anthony belong on the Banana Boat ? When engaging in these arguments, we typically use statistics like points or rebounds per game to back up our points, but at the end of the day, the groups are more or less kind of arbitrary. But what if there was a way to algorithmically sort observations into groups based on shared characteristics using machine learning methods? Enter clustering , which is the methodology of grouping similar observations into groups, or "clusters", using a mathematical distance metric derived from a set

Analyzing Hip Hop - Who's Most Lyrical, What Determines Popularity, and More

Have you ever thought about bringing cold, hard statistics to one of life's greatest artistic joys? Well fear not, because in our increasingly data-driven world, our analyst friends are hard at work attempting to statistisize (numerize?) everything you can think of, so we can analyze and therefore optimize it. One of the art realms that is increasingly falling under the purview of data science is music. We all benefit from it in the form of curated daily Spotify playlists and Pandora stations that allow us to find new artists and songs. I was recently able to get my hands on a Spotify dataset that contains data on over 160k tracks dating from 1921 through December 2020. Aside from containing some basic features like track name, duration, and release date, it also contains some advanced metrics as calculated by Spotify like "track positivity" (is it a sad, depressed song, or a happy, positive song?), "danceability", "energy", "speechiness" (

Statistics with Sasa

Search This Blog