Posts on Read Only Memory

My Story - Growth from IC3 to IC5

Mon, 06 Oct 2025 00:00:00 +0000

My first 3 years at Meta & career growth

It’s been a while since I last wrote here — work and life have been full, and writing fell off my routine. A lot has changed since then. Over the past three years, I’ve gone from an IC3 engineer to IC5 at Meta.

It’s been fast, chaotic at times, and full of learning curves.

This post isn’t advice — just a few honest notes on what that journey looked like for me.

IC3 — Starting Out

When I joined full-time, I had some context already from a previous internship. That helped a lot — I could ship code from day one and didn’t have to learn Meta’s development stack from scratch.

I joined the Checkout team as a backend engineer. About ten of us worked on backend, with another ten on the sister frontend team and a few PMs across the space. It was a big, fast-moving surface with a lot of cross-team dependencies.

The first few months were intense. I had to learn not just the product logic, but how the whole production system fit together — everything from loggers to metrics pipelines to how code made it into prod.

The “war room” atmosphere built a strange kind of camaraderie; everyone was in it together.

Projects

Meta Checkout “War Room” – supporting stabilization and reliability work during heavy launch cycles.
Meta Checkout Shopify SDK – owned the Meta-side API design and communication flow between Shopify and Meta (and got my first taste of cross-company collaboration).

IC3 was about execution. Shipping quickly, asking the right questions, and being reliable. Nothing fancy — just doing the work well.

IC3 → IC4 — Building Independence

It took about a year to go from IC3 to IC4.

The biggest shift wasn’t technical — it was how I approached work. At IC4, no one tells you what to do. You’re expected to find things that matter and push them forward yourself.

The moment that made a difference was when I took initiative to refactor a large part of our codebase. It wasn’t assigned — it was just something that had to be done if the team wanted to move faster. That project forced me to understand the system end to end and coordinate across engineers to avoid breaking things.

I was lucky to have strong mentors (IC5+) who taught me how to balance technical idealism with pragmatism. One funny twist: the project I spent the most time on that year never shipped — a strategic decision above my pay grade. But the promotion still happened, because what mattered wasn’t output, it was independence, initiative, and reliability.

IC4 — Broadening Scope

Soon after the promotion, I got reorged into the Orders team. I didn’t choose it — it just happened — but it turned out to be good for growth. I had to re-onboard from scratch and learn a completely new system.

I worked on the Order Management System, the backend that handles everything from payment capture to refunds.

Projects

Tax-Inclusive Pricing – a horizontal change that touched every order flow (refunds, fulfillment, capture). A lot of careful coordination.
BYOG (“Bring Your Own Gateway”) – building a new order flow to integrate external gateways like PayPal.

It was challenging at first. The new team had different norms, and I missed my old context. But adapting to change ended up being a skill in itself.

Over time, I realized I wanted to work on something more technical and foundational — closer to systems and infrastructure. Product cycles can be exciting, but also repetitive. I wanted to go deeper. That’s what led me to make the switch to AI Data Infrastructure.

Switching to AI Data Infra

Joining AI Data Infra was like starting over again. Different domain, different tools, different way of thinking.

But I was one of the first engineers on the team, which meant more ownership and less guardrail. I had to learn about distributed data systems, LLM training workflows, and what “data quality” means at scale.

Projects

GitHub PR Dataset – a research-oriented project to ingest and analyze open-source pull request data.
Code Data Sourcing – infrastructure for extracting and structuring code data for training models.

The ambiguity was high, but so was the sense of impact. I was no longer just building product features — I was helping build the data backbone for AI research.

It was easily the steepest learning curve of my career.

IC4 → IC5 — From Execution to Ownership

After about two years at IC4, I was promoted to IC5.

The expectations at this level are different. You’re not just delivering tasks — you’re creating direction.

I started mentoring other engineers, leading designs, and architecting new pipelines from scratch.

The GitHub PR dataset was especially tricky: open-ended, research-driven, and full of uncertainty. Progress depended on aligning people who didn’t technically report to me, finding clarity in fuzzy goals, and building something useful out of it.

At this stage, what mattered most was:

Consistency – sustained output over time.
Initiative – finding problems before someone assigns them.
Leadership in ambiguity – bringing clarity to undefined areas.
Time and trust – promotions at senior levels are built over multiple review cycles.
Impact (and some luck) – timing and organizational needs always play a role.

It wasn’t about one big launch. It was about building credibility, judgment, and calmness over a long stretch of work.

Looking Back

When I joined as an IC3, I thought growth meant learning new technologies.

Now I think it’s more about learning how to operate in uncertainty — how to keep moving when direction isn’t clear.

If I had to summarize:

IC3 taught me how to execute.
IC4 taught me how to take ownership.
IC5 taught me how to create direction.

Each step forced me to relearn what “being good at my job” means.

I’m grateful for the people and teams I’ve worked with — and for the luck of being in places where I could keep learning.

More reflections (and hopefully more regular writing) to come.

First 3 Months at Meta

Sun, 23 Oct 2022 00:00:00 +0000

Reflecting back at my first 3 months at Meta

It’s been over 3 months since I started my first job out of college - and I wanted to leave a blog post about it. A few things I’ll cover in this blog post: my bootcamp experience at Meta, and the onboarding process after team selection.

Bootcamp Experience

Most of you are probably wondering what I mean by bootcamp. Isn’t that a military thing where new recruits go through weeks of basic military training? Well yes, but that’s not what I’m going to write about here (thankfully).

Meta has a long-standing tradition of having all the new hires go through weeks of training (different training depending on roles), regardless of seniority. My understanding is that new hires that are recruited through the standard pipeline (meaning that you are not recruited for a specific team), need to go through bootcamp. Bootcamp serves largely three purposes:

Get accustomed to Meta’s work culture & company values.
Get hands-on experience with Meta’s internal tools.
Choose a team!

I found my time in bootcamp to be a little bit redundant, since I’m a returning intern. I was already relatively familiar with a lot of the internal tools, and had first-hand experience working on a real project last summer - so I already had a decent understanding of Meta’s fast-paced work culture. But it was a great few weeks to get myself mentally prepared to start working at full-pace, without having the pressure of having to deliver a project. I also thought that the bootcamp program is incredibly well structured. I was assigned a dedicated bootcamp mentor, and there were specific engineering trainings available (e.g. iOS, android, backend, etc).

The most exciting part of bootcamp is undoubtedly the team selection process. After the first few weeks, I started interacting with hiring managers that were looking to hire engineers for their team. I worked with ~8 teams and strategically narrowed down my team of choice. My guidelines for evaluating teams were:

Instinctive feeling from interacting with team members
Instinctive feeling from interacting with team manager
Team’s area of work (Do I feel excited about what the team is working on?)
Potential projects I would work on if I joined
Team’s culture around in-person work (I preferred joining a team with a decent amount of in-person work)
Team’s culture around mentorship

In retrospect, I think these guidelines formed a solid framework for my team selection process. It wasn’t too hard to narrow down to 3 teams - but making the final decision was a bit of a struggle. At the end, I opted to listen to my gut feelings. The best part about Meta’s bootcamp is that you can choose a team after working on small engineering tasks with the team (and also attend team meetings) - so you at least have some idea of what your day-to-day life would look like if you joined the team. At the end, I chose to join a team that owns the checkout backend & infrastructure across Meta’s Family of Apps. So far, I couldn’t be happier with the choice I made.

Team Onboarding & Learnings

Technically I’m still onboarding, but I feel like the most basic onboarding is behind me now. I still have to ask questions, of course, but I’m largely able to navigate through tasks assigned to me without too much help. Here’s what I learned from the last few months of onboarding as a newbie to the team.

Do your research before asking questions.
When you are truly stuck, don’t be afraid to ask questions to unblock yourself.
Read up on the documentations you can find about your team’s codebase.
Don’t overwhelm yourself by trying to understand the entire codebase in a few weeks. Start with understanding small parts of it, and gradually connect the dots.
Block time for reviewing code everyday.
Don’t back down from tasks that seem challenging. You can do it.
Get in the habit of creating documents when communicating with other engineers. It’s easier for them to follow if you have a document ready, rather than trying to explain everything on the spot.
Schedule regular 1:1 meetings with manager & mentors. I feel like this is by far the best way to ensure you are getting regular feedback (this is especially important for newbies).
Don’t be shy to ask for feedback. It’s always nice to hear that you are doing well, but the best of them all are the constructive ones - ask what you can do better to improve. How can you start operating at the next level?

So far it’s been a really exciting first few months at Meta. My team has been nothing short of amazing, and I’m working on an exciting project that I’m passionate about delivering. I knew I’d enjoy working at Meta because I had a great time last summer as an intern - but to be a full-time engineer is definitely a different feeling. I’m fortunate to wake up everyday, excited to work :)

Configuring Portable Dotfiles

Sat, 30 Jul 2022 00:00:00 +0000

A blog post on how to manage dotfiles with GNU Stow

I recently stumbled across a problem that I never really had to solve before. As I started my first full-time job, I received a new laptop for work-related use. I also set up my development server, which is a remote computer sitting somewhere in a data center that I can SSH into for development purposes. As my preferred choice of text editor is Neovim, I have a lot of configuration files that go along with it. I also have some custom tmux configurations. What about ZSH configurations? The list goes on. One way of doing it would be to manually copy-and-paste all the configuration files. But there had to be a better way - one that doesn’t require so much error-prone manual reptition. If I start a new job at a different company, or if I get a new personal computer for myself, I don’t want to have to continue to copy-and-paste all these “dotfiles”. So I searched for a way to make my dotfiles “portable”. The solution that I decided to go with was GNU Stow - in this post I’ll explain a little bit about how Stow works.

What is GNU Stow ?

Following is taken from the GNU Stow official website.

GNU Stow is a symlink farm manager which takes distinct packages of software and/or data located in separate directories on the filesystem, and makes them appear to be installed in the same place. For example, /usr/local/bin could contain symlinks to files within /usr/local/stow/emacs/bin, /usr/local/stow/perl/bin etc.

How it works

What makes GNU Stow so great is that it makes symlinking the config files a breeze. This makes it possible to simply have all of your dotfiles in a single VCS (Version Control Software) repository, whereever that may be in your machine. Let’s go over it step-by-step and see how it works.

1. Create a `.dotfiles` directory

In your home directory (not a requirement to be in your home directory, but this is how most people choose to do it), create a .dotfiles directory.

$ pwd
/Users/jeffyang

$ mkdir .dotfiles && cd .dotfiles

2. Create subdirectories for all the programs you want to configure through dotfiles

Now that we have a .dotfiles directory, we need to put create subdirectories for all the programs that we want to configure and manage with Stow. In this example, I’ll go ahead and create three subdirectories for nvim, and zsh.

$ pwd
/Users/jeffyang/.dotfiles

$ mkdir nvim zsh

$ tree
.
├── nvim
└── zsh

For each of the subdirectories we created, Stow will see these as individual worlds. Inside each of the subdirectories that we just created, we need to make sure that we maintain the exact same structure as home directory. For Stow, the directory structures we create for each of the subdirectories nvim, and zsh will be the symlink paths that it will create. Let me show you exactly what I mean through real examples.

For nvim, the configuration files would usually live in my home directory under {$HOME}/.config/nvim. For Stow to know exactly where to symlink the dotfiles to, we need to emulate this exact path.

$ pwd 
/Users/jeffyang/.dotfiles/nvim

$ tree
.
└── .config
    └── nvim
        ├── autoload
        │   └── plug.vim
        ├── init.vim
        ├── lua
        │   └── jeffyang
        │       ├── colors.lua
        │       ├── git-signs.lua
        │       ├── keymap.lua
        │       ├── lsp-config.lua
        │       ├── lua-line.lua
        │       ├── meta-lsp-config.lua
        │       ├── null-ls.lua
        │       ├── nvim-cmp.lua
        │       ├── telescope.lua
        │       └── utils.lua
        └── plugins
            ├── base.vim
            ├── neoformat.vim
            ├── nerdtree.vim
            ├── tabline.vim
            └── telescope.vim

The specific organization of my nvim config files are unimportant - what’s important to note here is that inside the nvim subdirectory we just created, I have .config/nvim/ which contains init.vim. This way, Stow can symlink the files under the nvim subdirectory correctly.

Let’s try another example with zsh. This will be a bit simpler, because ZSH configuration files typically live in {$HOME}/.zshrc. We’ll create this exact structure for Stow.

$ pwd 
/Users/jeffyang/.dotfiles/zsh

$ tree
.
└── .zshrc

Hopefully now you can see the pattern - the key is to keep the directory structure exactly the same as you would normally under your home directory. Now, let’s see how to actually use Stow to create the symlinks so that the configuration files all work as expected.

3. Stow’em all!

This is arguably the easiest part.

$ cd ~/.dotfiles
$ pwd
/Users/jeffyang/.dotfiles
$ stow nvim 
$ stow zsh

That’s it! Stow will create all the symlinks needed to ensure that your configuration files for all the programs that you want to mangae just work.

Conclusion

What’s so great about Stow is that you can keep all of your dotfiles in a single VCS repository. You can then push it up to Github or any other VCS cloud provider of your choice. Any time you want to set up your configuration files for the programs you care about, you simply need to download them from the cloud, and Stow them. As a software developer, I can’t appreciate Stow enough for finally bringing all my dotfiles under control :)

You can check out my dotfiles repo here.

Navigating a Computer Science Degree

Sat, 19 Mar 2022 00:00:00 +0000

Relection on my life as an undergraduate CS student and takeaways

I just completed my degree in computer science at Northwestern University. Coming into college, I hadn’t written a single line of code. Not even a line of “hello world”. I never saw myself majoring in a science/engineering field. Throughout my middle school and high school life, I always considered myself more gifted in things like public speech and debate - so naturally, I considered going into law, although it was more just a mere stream of thought than a decisive opinion. I came to Northwestern thinking that I’d study economics (because it seemed like the safe major to go with, for some reason) and international relations. 6 years and a half later (I took 2 years off due to mandatory military service in Korea and another half a year off due to COVID-19), here I am, with a degree in computer science and a full-time job at Meta (Facebook) secured. It wasn’t easy, but I truly had an amazing experience at Northwestern. In this blog post, I want to jot down a few notes on how I navigated my transformative adventure into the world of computer science and technology.

Disclaimer

Before we further dive into this post, I’d like to carefully just put this out there - I am an ambitious person. I see myself as an over-achiever, and I strive to be a better version of myself every single day. That’s just who I am. At the same time, I also acknowledge that not everyone is the same. Not everyone has the same values in life, and different people live with different principles in life. I respect that. This blog post is not meant to persuade readers that they should agree with my takeaways from college life. Even more so, this blog post is not meant to intimidate prospective computer science students. I am simply reflecting on my college life as a computer science (international) student who always dreamed of starting a career as a software engineer in the U.S. after graduation. For those who are still interested in reading on, I hope this post will help navigate your journey.

Computer Science is Not For Everyone

The number of computer science students grew exponentially in the past few years. As someone who has been in college since 2015, I’ve experienced this boom first-hand. When I first came to Northwestern, economics was by far the most popular major among undergraduates. Fast forward to 2022, computer science is one of the most popular majors, if not the most popular major. There are good reasons for this. Students seek to get a good return on their investment in college tuition, and a computer science degree can pay a handsome salary straight out of college. Let’s address the elephant in the room. In 2022, fresh grads with offers from top tech companies score a 150k+ salary. Many computer science students also pursue summer internships which can pay 20k+ over a single summer. Even though most of these top-paying jobs are in high cost of living areas (e.g. SF, NYC, Seattle), there is no denying that making 150k+ a year with zero years of experience is fantastic by all standards and measures. Add to that the glamorous picture the media has successfully painted around the tech industry - employees playing ping pong at work with free coffee - it’s no surprise we see a record number of computer science grads every year. However, as great as all this may sound, I cannot stress enough how important it is to seriously ask yourself if computer science is the right path for you before diving in headfirst. I can confidently say that computer science is not for everyone. Just a few off the top of my head: are you okay with spending 10 hours straight debugging a program? What if after 10 hours of debugging and losing sleep, you find out there was a missing semi-colon on line 37? Are you willing to spend time outside of class building projects that may never see the light of the world? Are you willing to constantly learn new technologies? These are some of the experiences you will go through if you decide to pursue a computer science degree and work in a related field. I can assure you that if you are simply in it for the money, you will get burnt out very quickly. It’s okay not to jump on the computer science hype-train just because everyone else seems to be jumping on it. I advise you to take a few introductory computer science classes to see if it’s something you’d like to continue with, before fully committing.

School Tips

Tip 1: Apply Yourself While at College

Apply yourself while at college. Either you are paying for it (and you have student loans), or your parents are paying for you. Either way, going to college is a massive investment - and you should try everything you can to make the most out of it. Apply yourself in all the classes and take advantage of the resources that make colleges different from bootcamps. Try to sit in the front and pay attention to the lecture. Be curious, and be express your desire to learn. You have extremely capable professors - go talk to them. Go to office hours and form a close relationship with one or two professors. These small things can make a huge difference - it will open doors to research opportunities, professors will be more willing to write letters of recommendation, and some of these professors may even have personal connections to startups that you may be able to intern at.

Tip 2: Utilize Free Online Resources

The internet has certainly changed the landscape of almost every industry, and education is no exception. There are so many free online resources that you can take advantage of. Although it’s crucial to go to your classes, don’t be afraid to learn on your own using resources that can easily be found online. Google is your friend. Youtube is also your friend. Take advantage of them! Also, top CS schools (e.g. Stanford, MIT, Carnegie Mellon) often upload their most popular courses on Youtube for the world to learn from. If you feel like this is something that you would benefit from, go for it! I personally found some of these resources extremely helpful in cementing my understanding of certain core CS topics.

Tip 3: Don’t Cheat

Cheating is quite a serious problem in higher education. Many students face disciplinary measures of varying degrees due to violation of academic integrity policies. As is the case with anything else, cheating in computer science classes won’t get you anywhere. You will be unable to learn the core concepts and materials that will be crucial to learning the next thing. Once you start “borrowing” code from your peers without even understanding it, you’ll find yourself falling behind quickly. It will lead to a vicious cycle of continuous cheating. In the best case, you will have graduated without getting caught - but you wouldn’t have learned anything while paying tens of thousands of dollars for your education. In the worst case - you could get kicked out of school. So don’t cheat. Try to actually learn the materials and go to office hours to ask questions. Consult the numerous free online resources that can help you learn outside of class. If you find yourself having to cheat over and over again in your computer science courses - this may be a good sign that you may not be the best fit for a CS degree.

Tip 4: Don’t Be Intimidated

I was very fortunate to have someone who cared about the learning environment as my very first computer science professor. It was a mandatory introductory computer science class at Northwestern - and there were students of varying coding abilities. Some students started coding at a much younger age - and there were students like me, who just started coding in college. The professor didn’t let the more advanced students show off and didn’t let them intimidate other students by asking questions about code compilers in front of a class just learning how to write a “hello world” program. But I do acknowledge that not everyone will be lucky enough to have such a caring professor! In that case, do not be intimidated by students that may seem to be ahead of you. Don’t let anyone make you feel small, or inadequate. As long as you are ready to passionately learn and put in the effort required, that gap between you and the student asking about code compilers will converge to zero over the next few years in college. Also, don’t feel too rushed to learn everything at once. If you are surrounded by students who seem much more advanced than you are, you may feel like you have to learn a million things at once. That will put a massive mental overhead on you. Just focus on perfectly mastering the class materials. You’ll see that learning becomes a lot easier once you are better equipped to learn the more advanced concepts. So don’t rush it!

Tip 5: Find a Mentor

Find a mentor if you can. It’s best to find a mentor that can interact with you in person - but if you can’t, at least find someone (e.g. Youtuber, Twitch tech streamer, etc) that you admire and respect. If you are lucky enough to find someone that you can personally interact with, you are in a great spot. What’s so great about having a mentor is that they are people who have already gone through what you are currently going through in the past. They have experience walking down the path that you are walking through right now. So mentors can offer you invaluable advice - but remember that you have to be proactive about it. No mentor will baby-sit you. You have to make sure that you do all the hard work, and seek guidance from the mentor as much as you can. I was fortunate to have a mentor, and I ended up getting an opportunity to intern at a startup that he founded. Mentors can also be a source of inspiration, as I looked up to him and aspired to become like him one day.

So then the question must be, how do you find a mentor? There is no easy answer for this. But I’d recommend attending tech-related club events and networking events to expand your network. Do you have school alums working at tech companies attending an info session? That’d be a great chance for you to get to expand your network. Go for it. Also, there can be upperclassmen in your computer science classes that may be willing to help you. Always keep an open mind, and be proactive about everything.

If you are still unable to find anyone or are simply too shy, you could also look for tech Youtubers or Twitch tech streamers. This would be a slightly different type of mentorship, as you probably won’t be interacting directly at a personal level with them. For example, I admire the passion that this Youtuber shows. He shares some personal stories as well as engaging videos about his journey in mastering Vim (the text editor). I found it extremely helpful to have a source of inspiration that motivated me to push myself and get better everyday.

Tip 6: Find a Hobby

It’s crucial to develop a sustainable study schedule. It can be extremely draining to constantly have to code and be under pressure. So instead of trying to force yourself to stick to a grueling study schedule that can quickly burn you out, create a sustainable pattern that can help you finish the years-long marathon. What that hobby may be really depends on what you like to do. While video gaming and binge-watching Netflix are certainly valid ones, I’d recommend that you do something more active, if possible! It’s important to take care of your physical health as well. For me, I try to hit the gym almost every day, and it helps me clear my mind.

Tip 7: It’s Okay to Take Time Off School

For those of you that are fortunate enough to be able to take time off college, it’s okay to do so! It’s not a race to graduate as early as possible. But this doesn’t mean that you should take time off just to rest. Try to make the most out of the time off school. I took 6 months off school to pursue an internship opportunity at a startup. While working, I could see how knowledge learned in school can be applied in the actual industry - this got me even more motivated to take certain classes once I got back to school. Don’t think of college as a 4-year race. If you can, it’s okay to take your time. Do it your way.

Tip 8: Start Learning About Money & Personal Finance Earlier

Start learning about money earlier. Get a credit card and start building your credit! Especially if you have an internship secured, you should start looking at investing early. Your time invested in the stock market is the single most important factor of being a successful investor in the future (if you are a long-term investor, like me). Since software engineering internships pay quite a decent salary (ranging from $20 ~ $60+ an hour, depending on the company), you can get a head start on your investing journey. I highly recommend The Psychology of Money - it’s a fantastic and easy read for people looking to gain control of their personal finance.

Tip 9: Explore Different Areas of Computer Science

At first, I was extremely confused by the generic term “software engineer”. It seemed to me that a person writing C code for a device driver would have a very different job than someone writing frontend javascript code to build UI. But they are all referred to as software engineers. Do I need to learn how to do everything from embedded systems to CSS to be hired as a software engineer? The answer is, no. There are many different fields of software engineering (e.g. Web Frontend, Mobile Frontend, Backend API Dev, Systems, Machine Learning, QA Engineer, Networking Engineer, etc) and you are most likely going to have to choose one or two fields that you are interested in exploring further. As an undergraduate student, you certainly do not have to feel pressured to specialize in any of these fields - simply understanding that there are many different areas of software engineering is good enough. As you go through your courses in college, try to see which classes and technologies interest you the most. Another great way of finding out about which part of the “tech stack” you would be a great fit for is to work on side projects. For example, if you are building a relatively standard web application with a separate frontend and backend, you should be able to get a feeling of what it would be like to work as a frontend engineer / backend engineer. If you decide to get more complicated with your project and add fancy bits of infrastructure to it by using AWS, Docker, and many other technologies, you will also be able to get some exposure to deeper backend infrastructure work. Little by little, explore different areas of software engineering and see what you find interesting.

Tip 10: Get Extremely Proficient with at Least One Programming Language

Become extremely proficient with at least one programming language! You’ll be surprised to see how many students graduate with a computer science degree without having a single programming language they are extremely comfortable with. The reason why you should have at least one programming language that you are extremely proficient in is very simple: it makes learning other languages that much easier. Different languages have different syntax, but a lot of the languages share similar concepts. As a software engineer, you will have to learn different languages fairly quickly on the job, and that becomes a lot easier if you have already mastered a language before. Another reason is that you’ll have to interview to get jobs, and it helps to be very comfortable with a specific language of your choice. It’s much easier to focus on the logic of the code (i.e. the problem that you are trying to solve) if you are confident about your ability to actually write the code out in the language of your choice.

(Highly Opinionated) Tip 11: Learn How to Use Vim

Learn how to use VIM! I use Neovim as my primary text editor, and I personally think it has boosted my productivity. As a software developer, you’ll spend a lot of time in the terminal, so I think it’s great to get used to tools that live in the terminal, rather than depending on GUIs. There’s a steep learning curve to learning how to use Vim, but I’d say it’s totally worth it :)

Recruiting Tips

Tip 0: Know Your Data Structures & Algorithms (feat. Leetcode)

Yes, this is tip #0. I cannot emphasize enough how important this is if you want to get a job in software engineering. Leetcode-style problems involving knowledge of data structures & algorithms is the industry standard way of conducting technical interviews. Like it or not, you are going to have to get good at it. It may seem daunting at first, but the key is to create an interview prep plan and stick to it. It’s extremely important not to rush it. Solving 2 problems every day consistently is far better than trying to solve 10 questions a day for two weeks. Try to learn the key concepts and patterns instead of memorizing the questions. There are many different sites out there that let you practice these kinds of problems, but my recommendation is to stick with Leetcode. They have by far the largest number of questions and a very active community. I’d say that purchasing their premium plan is definitely worth it if you are in the Leetcode grind. Not sure how to get the most out of Leetcode? Look no further than [this post](https://leetcode.com/discuss/career/450215/How-to-use-LeetCode-to-help-yourself-efficiently-and-effectively-(for-beginners).

When you are closer to the actual interviews, make sure to do multiple rounds of mock interviews before the real one. Pramp.com is a great site to brush up on your interview skills.

Tip 1: Think About Internships Early in College

If you are considering hopping right into the job market after graduation, the fastest & the most straightforward way of achieving that goal is through scoring internships while a student - the most important one being your junior year internship. Typically, students are able to get the “big” internships in their junior year - but there’s no harm in starting early. Throw yourself out there in your freshman/sophomore year. There are internship programs from big companies like Google STEP and Facebook University that are specifically designed for 1st and 2nd-year students. You can also apply to startups on sites like angel.co. Don’t be afraid to reach out to people on LinkedIn to ask for advice and/or potential internship opportunities. Just remember to be respectful of them. If you start thinking about internships early in your college life, you’ll be well equipped come junior year when the stakes are high. You’ll know how recruiting works in general, you’ll likely have a few recruiters that you’ve already established connections with, and the whole process won’t feel so daunting and foreign.

Tip 2: Do Side Projects

I’m sure you’ve already heard someone say this to you, but I’m just gonna say it again. Side projects are extremely important as a college student trying to break into the industry without much work experience. If you already have multiple internships under your belt, then side projects may be less important. However, for those of you that are looking for that first internship and have no proper job experience to show for, side projects are critical. Why? Employers want to see that they are hiring someone who can contribute without them having to spoon-feed everything. In software engineering, learning computer science theory and reading thick textbooks isn’t good enough as it doesn’t necessarily translate into practical skills. Side projects outside of class is a great way to bridge that gap between knowledge learned in class and practical skills required to work in the industry. For example, side projects can teach you a lot about how the Git version control software works, which is pretty much what the entire industry uses. You can also gain exposure to popular web frameworks such as React and Vue.js for frontend and Django and Express.js for backend. If you decide to get fancier, you could also play around with various cloud computing platforms such as AWS (Amazon Web Services) and GCP (Google Cloud Platform). Knowing how to use these tools makes you a much more attractive candidate in the job market.

As a college student, you are a part of your local community (of your peers and neighbors). As a student developer, you are in a unique position to try and solve some of the problems that are solvable through technology. For example, I built a website that attempts to make off-campus room search easier - an exclusive Northwestern marketplace for sublets. Although the website isn’t currently actively used, I learned a lot while building this website with my team. I also talked about this project in every single interview. The reason why I recommend trying to solve a problem that your community uniquely has is that you can keep yourself motivated to actually finish the project. Aim to launch the project - it’ll be a cool experience!

Tip 3: Write a Proper Resume

Having a resume that is succinct and clear is incredibly important. Obviously, the more quality content you have to put on your resume, the better it is. However, writing a good resume is more than just having good content. Make sure that you have your resume formatted properly. Make sure there aren’t any silly grammar mistakes. Make sure you are putting things in the form of accomplished [x], measured by [y], by doing [z]. I found this guide to be particularly helpful. Remember that recruiters don’t look at your resume for more than a few seconds - it’s up to you to create a resume that’s impressive and clear enough to ensure that your resume doesn’t end up in that “no pass” pile.

Tip 4: Try to Contribute to Open Source Projects

Try to contribute to open-source projects. Not only can you learn about how software development is done in larger companies (e.g. code reviews), you can also learn the basics of Git - these are extremely important skills if you want to be successful on the job. On top of that, contributing to open source projects (especially the more famous ones) can make your resume really stand out. If you are interested but don’t know how to get started, this is a great place to start.

Tip 5: Participate in Hackathons

To be truthful, I’ve only participated in one Hackathon in my college life. I wish I had gone to more. I had such a great experience at Stanford’s TreeHacks - I had the chance to meet numerous like-minded people who were passionate about using technology to make the world a better place. Participating in a Hackathon can be a great way to pad your resume with an amazing project. Go and have fun hacking over a weekend!

Tip 6: Take Advantage of Networking Events on Campus

Take advantage of on-campus networking events as much as possible. While networking may not be as important for computer science students compared to some of the other industries, it still can make the world’s difference. The most difficult part of the recruiting process is often landing the interview itself. By attending on-campus networking events, you can certainly maximize the chances that you’ll at least get an interview. Unlike when you apply to a company online, where your resume will be sitting with tens of thousands of other resumes, you can pass you resume directly to a campus recruiter during these events. Some companies even conduct on-campus interviews as well. Don’t be shy and get yourself out there!

Tip 7: Don’t Compare Yourself to Your Peers

Recruiting is, for most, a draining experience. There are many different reasons as to why it can be mentally draining, but I think the single most unhealthy thing to do is to constantly compare yourself to your peers. If you have friends around you landing offers from top companies while you are struggling, it can be tough. You’ll also see that LinkedIn explodes with posts from people who are posting their success stories - about their dream FAANG offers. It’s important not to compare yourself against them. Remember to go at your own pace, and that you only need one offer no matter how many rejections you get. If you feel down, just take a look at this website, and see for yourself how much luck is involved in the process. You just need to keep going and never give up :)

Tip 8: Getting Ghosted is Okay

During my first recruiting season, I applied to about 200 different companies and was ghosted by more than 100 of them. This is normal - especially if your resume doesn’t have big-name companies on it yet. Don’t take it personally, and focus on the companies that do get back to you. Recruiting is a numbers game. During the recruiting season where I landed offers from Microsoft and Facebook (this was my second recruiting season), I applied to about 200 companies, heard back from 15, had final round interviews with 5, and received 2 offers.

If you read along this far, thank you! I wanted to share my experience studying computer science in the hopes of helping someone out there. Best of luck in your journey!

A Practical Application of Redis Cache

Sat, 18 Dec 2021 00:00:00 +0000

How a Redis cache can help solve problems when fetching data

Recently, I’ve been working on a side project - a web application (inretrospect.finance) that lets users record their investments with notes for future reference. The whole project is built on top of a fantastic API from Alphavantage that provides detailed financial data on public companies. While it would have been great if it were possible to make unlimited API requests - their free tier plan only allows for 5 requests per minute and 500 requests per day. This obviously is a significant bottleneck, so I needed to find a solution. After putting some thought into this, I decided to cache API requests so that whenver there’s a cache hit, a request to the actual API endpoint is not made, therefore cutting down the number of requests to the API service. With that context in mind, let’s dive in.

1. What is Redis?

Redis (REmote DIctionary Server) is an advanced NoSQL key-value data store that is open source (BSD license) - often used as a database, cache, and message broker. Redis is increasingly a popular choice by developers that are looking for an extremely fast and versatile data storage. While it can serve a variety of different purposes, we will focus soley on using Redis as a cache - explicitly to speed up future requests for the same data. I am by no means an expert at Redis, and in this blog post my aim is to document my process of solving a relatively simple problem I faced while building a side project - not to give a detailed technical introduction to Redis. For that, I’d recommend visiting their official documentation.

2. Installing Redis

To get started, we first need to install Redis. I have a machine that runs MacOS, so please take that into consideration.

$ brew update
$ brew install redis
$ brew services start redis
$ redis-cli ping
> PONG

If you want to stop Redis from running in the background, simply run the following:

$ brew services stop redis

That’s about it - you should be able to run Redis on your machine now. For installing Redis on different operating systems such as Linux and Windows, I found this article to be helpful.

3. How can Redis Help?

Before we proceed, I’d like to explain a little more about the problem I faced, and why Redis turned out to be a great option to address the problem at hand. In the web application inretrospect.finance, I needed historical stock market data for stocks of user’s choice (US stocks). For the purposes of this application, I did not need real-time market data, but I did need the data to be updated once a day after market close. For example, if a user registered Facebook and Google as stocks of interest, I needed historical market data on both of these stocks in order to render a line chart. To help your understanding, below is a screenshot of the UI that needs such historical market data to render properly.

I obviously cannot magically pull stock market data out of nowhere - I needed to use an API that provides such market data. Now, let us think about how one might go about achieving this relatively simple task. The most naive and simple solution would be to send a request to the API endpoint whenever the UI is rendered. I used React to build the client side of the application - let us see what such an implementation would look like in code.

  // StockCard component
  const [stockData, setStockData] = useState(null);

  const fetchData = async () => {
    // API request is made here with the ticker symbol of interest
    try {
      const response = await api(`/stockData?symbol=${tickerSymbol}`);
      setStockData[response];
    } catch (e) {
      // handle error here
      console.log(e);
    }
  };

  useEffect(() => {
    // fetch data on every render of component
    fetchData();
  }, []);

It’s a standard API call that would occur in the useEffect hook of a React functional component - which updates the state upon a successful retrieval of data from the API request. This implementation works, but what would happen if someone had 5 stocks registered and hit the refresh button 3 times? That would be 15 API calls in the span of 5 seconds - and this is problematic in two ways. Firstly, the free tier API service that I used only supports up to 5 API calls per minute. Secondly, this naive implementation would be suboptimal in terms of user experience, since every single render of the component would need to wait for the API to return data. This is where caching can help us significantly. If we can cache the data returned by the API somewhere, we would be able to fetch data that has been requested previously right from the cache instead of waiting for the API to respond. That’s exactly where Redis comes to the rescue!

3. Setting up Redis Within an Express Application

In order to handle the caching logic, I created an API endpoint that acts as a wrapper to the actual API endpoint that provides the market data. I did this using a Node.js Express application - let us see how to set up Redis within a basic and simple Express application. Since this post is not meant to be an Express tutorial, I’ll assume that you already have an Express app up and running, or that you’d be able to set it up yourself (it should be rather simple).

We first need to install Redis client that will help us interact with the Redis service that is being run in the background. Because our backend is being built in Node.js, we will need to do the following within the Express project directory:

$ npm install redis
OR
$ yarn add redis

The redis package installed will help us interact with Redis instances easily in our Node.js codebase. Let us first create a Redis client in our Express application before moving on to the actual business logic (i.e. caching API responses).

// load dotenv file for sensitive information
require("dotenv").config();

const express = require("express");
const axios = require("axios");
const redis = require("redis");
const cors = require("cors");
const app = express();

app.use(cors());

const redisPort = 6379;

let client;
if (process.env.REDIS_URL) {
  // production
  client = redis.createClient({ url: process.env.REDIS_URL });
} else {
  // dev
  client = redis.createClient({ port: redisPort });
}

Note that I am initializing a Redis client in two different ways - one with a specified URL and one with a specified port number. I conscioulsy made this choice because I needed to configure Redis in Heroku. In practice, these two different ways of creating a Redis client bear no difference.

Now, let us see the code that actually handles the core logic. The flow is as follows:

With that in mind, let’s take a look at the code.

// implement API endpoint
app.get("/stockData", async (req, res) => {
  const tickerSymbol = req.query.symbol;
  try {
    client.get(tickerSymbol, async (err, stockData) => {
      if (stockData) {
        // cache hit --> simply return data stored in cache
        res.status(200).send({
          data: JSON.parse(stockData),
        });
      } else {
        // cache miss --> attempt API call
        try {
          const stockData = await axios.get(
            `https://www.alphavantage.co/query?function=TIME_SERIES_DAILY_ADJUSTED&symbol=${tickerSymbol}&outputsize=full&apikey=${process.env.ALPHAVANTAGE_API_KEY}`
          );
          // stringify data to store in redis cache
          const storableStockData = JSON.stringify(stockData.data);
          // store data from API in redis cache - expiration timeout 24H
          client.setex(tickerSymbol, 24 * 60 * 60, storableStockData);
          res.status(200).send({
            data: stockData.data,
          });
        } catch (err) {
          res.status(500).send({ message: "API Response Err" })
        }
      }
    });
  } catch (err) {
    res.status(500).send({ message: err.message });
  }
});

Relatively simple and straightforward implementation, right? Redis has an extremely friendly API and makes for a delightful developer experience. With minimal amount of code, you can bring drastic changes to the logic of your code for the better. If you want to check the Redis cache yourself, you can do the following:

$ redis-cli
// a cache miss
127.0.0.1:6379> GET "GOOG"
(nil)
// a cache hit
127.0.0.1:6379> GET "GOOG"
<DATA SHOULD BE SHOWN>

Next, let us check if Redis actually helped us achieve our goals - how much faster is Redis compared to an API call over the network?

4. Measuring Performance with Redis Implementation

It’d be interesting to see how much of a difference Redis can make - how much faster is it with Redis? To test this, let’s first try requesting data over the API that is not stored in cache.

// verify that there is no data stored
127.0.0.1:6379> GET "NFLX"
(nil)

Using Chrome Developer Tools, we can inspect the intricate details of networking operations. The screenshot below shows how long it took for the API request to process. “Waiting (TTFB)” indicates total time for the sent request to get to the destination (in this case the Express backend), then for the destination to process the request, and finally for the response to traverse the networks back to the client. It took 7.5 seconds for an API call asking for historical market data on Netflix stocks.

API call without cache.

Now, let us verify that data on Netflix is actually stored in our Redis cache.

// verify that there is data stored stored
127.0.0.1:6379> GET "NFLX"
<HUGE DATA LOAD STORED AS STRING>

Then, let’s refresh the page to initiate another call to our backend, and inspect with Chrome Developer Tools.

API call with cache.

Wow. The cached response only took 60 milliseconds. We just shaved off 99.3% of the total response time with the Redis cache implementation. It’s not only that - by caching API responses in Redis cache, I was also able to work with the Alphavantage free tier API during development (maximum of 5 requests per minute and 500 requests per day). If inretrospective.finance scales up in the future, there will definitely be a need to start using the paid plans, but for now, this will do.

Wrapping Up

What we went through today is an example of how the Redis cache can be used in a real-world project. Redis cache is so simple to use, yet so powerful. As developers, it is our responsibility to ensure that users get the best experience from the software we create - tools like Redis make our lives so much easier! I plan on writing another blog post on things I learned while building inretrospect.finance, so stay tuned :)

My Facebook Internship

Wed, 08 Sep 2021 00:00:00 +0000

Working at Facebook as a remote software engineering intern

I started my journey in computer science as a freshman in college. Little did I know, this journey would take me to the heart of Silicon Valley.

In this blog post, I share my experience as a software engineer intern at Facebook. For those who are trying to gauge if Facebook is the right fit - I hope this post will provide valuable insight. For those who are on the grind for an internship - I hope this will serve as a source of inspiration to keep going. Facebook seemed like a pipe dream, but through grit and little bit of luck, I’m really happy to have made the impossible possible.

Application Process

In tech, the greatest challenge is landing the interview itself. The competition for a software engineering gig is borderline ridiculous, with tens of thousands of students competing for only a handful of available spots. A LinkedIn resume drop or applying through a portal will only get you so far, and many times, without an interview.

My recommendation is to at least submit your application alongside a referral from a current employee of that company. For anyone applying to Facebook, I would highly recommend asking for a referral from someone who can speak to your qualities as an engineer.

However, my journey with Facebook started out a bit differently — I got lucky. A recruiter reached out to me on Piazza, a learning management site used by professors at my university, after viewing my resume. Frankly, it was the unlikeliest of all places for a Facebook recruiter to be reaching out. But it happened. I was invited to interview — although my first attempt at cracking the coding interview was unsuccessful.

Exactly a year later, in Fall 2020, I gathered the courage to contact the same recruiter and was lucky to be granted a chance to interview again. This time, I cracked both rounds of technical interviews. A couple days after the second round, I received a call from the recruiter with a formal offer to join Facebook for Summer 2021 as a software engineer intern. It felt surreal.

Recruiter message on Piazza Careers.

Team Selection

The team selection process for interns at Facebook is done in a manner so that both parties — the interns and company — benefit from the program. Few months prior to the internship, interns are given a survey through which they can express interest in certain pillars within Facebook.

Specific teams are NOT provided as options to select from, but interns can express interest in certain fields (e.g. machine learning, frontend engineering, backend engineering, etc) and app pillars (e.g. Instagram, Facebook, Oculus, WhatsApp, Messenger, etc).

Once the surveys are submitted, a mysterious script magically performs team matching. From what I experienced and heard from others, the team matching script does a pretty great job of satisfying both parties.

I was matched with the Business Messaging Lead Gen Ads team, a business-facing ads product team. It is a product used by businesses worldwide, and at the time I was matched, I felt it was an assignment I could excel in.

A Remote Internship

Due to the pandemic, the 2021 Summer Internship Program was fully remote. Although my team was based in Menlo Park, I chose to work from Los Angeles. It made the most sense as LA shares the same time zone, has more affordable rent, and offers a lot as a city, including outdoor hikes, beautiful beaches, and great restaurants — especially Korean ones. My roommate was also working remotely as a software engineer intern at a different tech company, so we decided to rent an Airbnb together.

Beautiful scenery of LA.

Working remotely meant that Facebook shipped its tech equipment to all interns.

Facebook tech equipments.

Facebook also sent out intern swag a few weeks before the internship!

Facebook swag.

I set up a nice home office for myself — having that ultrawide monitor was extremely helpful. 😇

Home office setup due to COVID-19.

Imposter Syndrome

Imposter syndrome is defined as doubting your abilities and feeling like a fraud. I heard and read about it on countless online forums, but had never experienced it myself until my internship with Facebook. As excited as I was to work at Facebook, the thought of working at one of the world’s leading tech firms was incredibly daunting. I always had the idea that engineers working at companies like Facebook and Google were geniuses — I never thought I would belong to one of them. As the start date loomed, I became slightly overwhelmed. Would I be able to meet expectations at Facebook? What if it takes too long for me to learn the internal tools that are required for work? What if other engineers on the team think I don’t deserve to be working with them? I spent hours reading about imposter syndrome on various tech forums online, and it offered a sense of comfort to know that many others were experiencing the same thing I was. I read a lot of advice along the lines of “There’s a reason why they hired you. Believe in yourself” — but still, I wasn’t convinced.

Ramping Up

In the blink of an eye, I started my internship. The first days consisted of orientation and general training on source control (Facebook uses Mercurial instead of Git). Once I completed general training, I held my first official meeting with my intern manager. We spent that time getting to know each other before diving into the details of the internship. My manager prepared a very detailed document on my intern project, including general onboarding documents on our team’s codebase. We also went over my goals and KPIs, ensuring all expectations were clearly communicated. This included preferred ways of receiving feedback, working hours, and more. I felt very supported by my intern manager throughout the internship, and the project was most definitely doable within the given time frame (it was well scoped out by my manager beforehand). I outlined 2 goals for myself and communicated these very clearly to my manager:

To receive a return offer at the end of the internship
To get to the level where I could confidently review another team member’s code and approve it (at Facebook, only one person needs to review and approve code to ship it to production.

I was assigned starter tasks that provided just the right amount of exposure to Facebook’s vast codebase. After completing the starter tasks in a few days, I moved on to tackling the intern project milestones. By this point, I felt increasingly confident that I would be able to handle the work assigned. It also helped to have a supportive team and intern manager who genuinely wanted me to succeed.

Not Just an Intern

At Facebook, I never felt like “just an intern.” I had full access to the gigantic monorepo (yes, Facebook keeps all its code in a single repository), attended meetings with full time engineers, performed code reviews, provided feedback on our team’s products, and had full control over my working schedule.

Furthermore, I worked on a full-stack project using technologies such as React for frontend and Hack/PHP for backend logic (Note: a lot of Facebook’s backend code is written in Hack/PHP, especially for product teams). Although I had extensive experience with React, I never worked with Hack/PHP, but it was an easy learning curve as it is quite similar to other programming languages I was already familiar with. As such, I finished my intern project in 7 weeks.

For the remaining 5 weeks, I drafted a proposal for my own mini-venture to build on the project I had completed. Once I received my team’s blessings, I collaborated closely with our product designer, product manager, and content designer to bring it to life.

That’s why what I loved about interning at Facebook was contributing to projects that impact the real world. The feature I built, for example, was highly requested by high-profile clients such as AT&T and Lexus.

Takeaways

I learned so much during my time at Facebook. Not only that, I left Facebook a much more confident engineer. Here are some of the takeaways:

You don’t need to be a programming prodigy to work at Facebook. You need to be self-driven and eager to learn!
Soft skills are extremely important for engineers. Code is merely the end product that software engineers produce, but a lot more goes into making that happen.
There are lots of internal tools to navigate at big tech companies like Facebook. Learning how to use them will help you become much more productive.
Don’t take existing code for granted. Just because it passed code reviews and was merged to the main trunk does not mean that the code is failproof.
Take advantage of the opportunity to work with senior engineers and learn from their code. Review their code and see how they write theirs differently. How would you have implemented the same thing yourself and how did they do it differently?

Tips for Future Facebook Interns

Communication is key - always make sure that your intern manager is on the same page as you. Share your progress daily, and whenever you reach an important milestone, communicate your accomplishment to your team as well. Communication was the single most important factor in having completed my remote internship successfully. It makes it easier for your manager to vouch for you in the final calibration as they’ll have a solid idea of your hard work. Two peer engineers also write reviews for you in the final calibration, so make sure to also communicate with them as much as possible.
Don’t be afraid to ask questions, especially when you are having trouble getting unblocked.
If you want to maximize your chance of a return offer, do yourself a favor and start working on your project as early as you can.
Once you receive comments on your code, act on it, and make sure to keep a checklist before submitting your code for review in the future. It’s best that other engineers don’t have to point out the same things over and over again.
Remember that an internship is a sprint, not a marathon. You only have ~10 weeks to prove that you can provide value to the team. Don’t focus too much on maintaining a great work life balance during your internship - you can worry about that when you’re full-time. Make sure you stay focused on doing whatever you can to deliver your project in time!
Own your project. Remember that Facebook treats you as a full-time employee during your internship. If you have ideas, share it with your team. Feel free to expand on the scope of your intern project. Showing initiative is a quality they look for - so don’t be shy!
Before asking design questions about the code you are writing, always form your own opinion first. They will be curious on what your thoughts are before they present their take on it. It’s always a good idea to outline your thought process before asking for an answer.

Working at an Early Stage Startup (Pt. 2)

Sat, 13 Mar 2021 00:00:00 +0000

What I learned working at an early stage startup - the engineering side

Disclaimer This post does not in any way represent the views of Fumi as a company, but is solely my individual opinion and reflection on my experience as a software engineer at an early-stage startup

I was a software engineering intern, after all - so let’s talk about things from an engineer’s perspective. To give you context, I was part of a small engineering team of 3, and did not have a direct report (i.e. no “manager”). Although my title was a “software engineering intern” I was more like a full-time contractor. I was (for the most part) expected to do things on my own, as the other 2 engineers did not have enough time to be taking care of an intern.

What did I work on?

I was a full-stack engineer - I worked both on the frontend and the backend of Fumi’s main product and Fumi’s back office application (inventory tracking, delivery tracking, etc). The Fumi mobile app used Ionic, Capacitor, Angular 9 on the frontend and used express, GraphQL and MySQL on the backend. The back office application used Angular 9 and KendoUI on the frontend, and used the identical set of backend technologies as the Fumi app.

Struggle 1 : Development Speed vs Quality of Code

The greatest struggle that I had while working at Fumi as an engineer was trying to figure out the right balance between the speed of development vs the quality of development. Every sprint, the COO pushed for a rather tight (?) schedule that was often unrealistic. Of course, it was up to us as developers to agree/disagree with the COO in setting the deadlines, but it was sometimes almost as if the company leadership really didn’t care about the quality of the software we were building. They did care about the product that the end-users will see, but not so much the internal software. It was a real struggle because as an engineer, I wanted to write and produce code that I would be happy to revisit in 3 months, but with the timeframe we were working with (runway of about 1.5 yrs), asking for more time to refactor and write maintainable, clean code was often difficult. Even writing proper test cases to increase confidence in code (especially during code release) was a luxury we could not afford. At the same time, I also understood that quick iterations (depending on user feedback) were absolutely necessary to survive as a startup. After all, what’s the point of writing 100% coverage test suites and meticulously refactoring code if the newly developed feature is going to be tossed out the window in a few weeks upon being labeled a “failure”?
The conclusion I walked away with was that it’s important to carefully evaluate whether or not certain parts of the code base are worth investing time into refactoring and perfecting. Is this piece of code an integral part of the user experience, and is the code likely to be used elsewhere in the future? If so, go ahead and ask for more time, and perfect the code to your heart’s content. Is not, feel comfortable moving on without looking back. An early-stage startup need not hold their engineering standards on the same level as Google’s or NASA’s - different kinds of engineering is required for different occasions.

Struggle 2 : Lack of Mentorship

As I mentioned above, I was more of a full-time contractor than an intern while working at Fumi. There was no structured program to help a junior engineer succeed and get guidance. This is very unlike interning at a larger company with a well-structured internship, where there is a designated intern manager and a whole team of engineers that is always willing to help out. Being able to figure things out on yourself (of course, with the help of Stackoverflow) is considered an important quality to have as a software engineer, but mentorship and guidance from experienced engineers is always invaluable. This is NOT to say that I didn’t learn from working at Fumi - I learned a lot. However, most of the learning came from teaching myself new technologies and having to do things myself, not from dedicated mentors.

Struggle 3 : Getting Used to Bad Engineering Practices

I am a firm believer that different kinds of companies require different kinds of engineering. For example, an engineering team at an early stage startup that is constantly fighting for survival cannot have the same level of attention to detail as an engineering team at Google can afford to have. However, if your goal is to learn the “best engineering practices” such as having thorough code reviews and 100% test coverage, an early stage startup may not be what you are looking for. Always chasing deadlines after deadlines, engineers at early stage startups are often forced to produce duplicate code, spaghetti code (that would be an absolute nightmare to refactor in 3 months), and hacky code - that just works. Most early stage startups don’t have the luxury to think months in advance and what to do when the company scales - shipping this feature on that date is what matters.

Struggle 4 : Communication

I want to stress the importance of communication. Communication with a product manager, a product designer, or company leadership is extremely important to keep everyone on the same page. For example, I found it very helpful to create slack channels or project proposal documents. It was also very important to learn to clearly communicate what I thought was an appropriate timeline for a given project. What I struggled with especially towards the beginning of my internship was to accept all the deadlines given. If you think a deadline is unreasonable, speak out and let yourself be heard! I realized that it’s very important to get everyone on the same page so that it’s easier to anticipate when certain features will actually be shipped to production.

Positive 1 : Shaping Engineering Culture

What was great about working at an early stage startup was that I could directly contribute to the engineering culture. I was able to persuade my team to set up certain code style guides, start a Notion documentation page on critical frontend and backend services, and also write a detailed onboarding documentation for new hires. Yes, my team was in its very early stages, so I was able to start these initiatives and actually have a say on how our engineering team should function. I’m pretty sure that this would not have been possible at a more established company, so it was a pretty cool feeling to be able to have this kind of influence, to say the least.

Positive 2 : Experience Working Across the Stack

Angular, GraphQL API Endpoints, MySQL DB Modeling, Jenkins build - these are some of the things I worked on over the last 4 months. It almost sounds like an entire engineering team’s job, right? Frontend, backend, and even a sprinkle of devOps. The reality at a very early stage startup with 3 engineers is that they cannot afford to hire specialists for every stack - all the engineers are generalists and are expected to fill in wherever they can. Hence, even as an intern, I had the opportunity to work across the entire stack, which in my opinion was a beneficial experience for me. Instead of being stuck with a small part of the frontend or the backend, I was able to draw out the frontend UI directly from Figma files that the designer gave me, implement the frontend logic, create necessary API endpoints, create a new MySQL database table if needed for a new feature, and then watch the Jenkins build succeed (or sometimes fail). I was able to see the bigger picture of how products are built and deployed, and I’m pretty sure it would not have been the case had I worked for a large company.

Positive 3 : Feel Like a Core Part of the Team

Last but not the least, I would highly recommend people to try working for a startup because it’s such a great feeling to feel like a core part of the team. At a large company, people come and go, and most people don’t feel like they are an absolutely integral part of the company. At an early stage startup with very few engineers, every single person on the team is an invaluable asset. Your opinion can have a direct impact on the product, and the all the other teams count on the engineering team to make things happen - and you get to be a part of it. I found that to be an amazing feeling.

What’s next for me?

Working at Fumi was definitely a great experience for me, and I certainly learned a lot. Not only did I grow as an engineer, but I also learned a lot about how startups operate. What’s next for me? I am extremely lucky to be able to say that I will be working as an intern at Facebook HQ this coming summer. Here are some things I am looking forward to learning more about (very generic because I don’t know which team I’ll be working with yet):

How are code reviews done at a large tech company?
How do senior engineers at a company like Facebook write code differently?
What kinds of internal tools do Facebook engineers use, and are they useful?
Am I good enough to be held against the highest standards in the industry?

I’ll definitely write about my Facebook internship experience at the end of this summer, so stay tuned!

Working at an Early Stage Startup (Pt. 1)

Sun, 28 Feb 2021 00:00:00 +0000

What I learned working at an early stage startup - the business side

COVID-19 brought an abrupt stop to my studies in the United States, leading me back to Seoul. As I was always interested in the startup scene, I could not turn down the opportunity to intern at an early stage startup when it presented itself. Following rounds of interviews, I landed an internship at Fumi, a Korean e-commerce startup catered to the senior fashion market. Fumi had recently raised $3M in Series A funding and was rapidly growing its team. Over the past 4 months as a software engineering intern at Fumi, I was able to grow not only as a software engineer, but also as a future entrepreneur. In this blog post I would like to share my takeaways from working at an early stage startup as a software engineer.

Business Side of Things

One of the best things about working at Fumi was that I got to experience the business and engineering verticals of an early stage startup. More specifically, I worked with the product team, which was composed of 3 software engineers, 1 product designer, and the Chief Operations Officer (who also largely took on the role of a product manager). Most of my business-side engagements occurred during daily product team meetings where we shared our progress and shaped short & long term goals for the product we were building. It was particularly interesting to gain insight into the company’s finance / product metrics on a daily basis. Here are some of the insights I walked away with following my internship:

Competition is Fierce in the Business of E-commerce

E-commerce is a highly competitive industry, especially in South Korea. There are extremely popular e-commerce platforms such as Coupang that already dominate the market (think Amazon). Coupang offers next-day delivery, one-click pay, and an expansive product lineup. In other words, as an e-commerce platform, it is very difficult to compete with such a massive and established entity. Even if we hone in on a more specific vertical like “fashion e-commerce,” the situation doesn’t get much better. This area is hyper-competitive in Korea because fashion is HUGE here (especially among the younger generations). Hence, there are apps like Musinsa, Ably, and StyleShare that are catered towards younger audiences and boast millions of active users. But what if we narrowed down the market even further - to senior fashion e-commerce, for example? All of a sudden there isn’t a clear market leader - and that is how Fumi came to be. The founder of Fumi saw significant market opportunity in the space of senior fashion e-commerce.

Market for Seniors is Big but Difficult

Fumi’s standout point in comparison to other e-commerce platforms was that we targeted senior customers. This is how Fumi was able to attract heavy-weight investors including Altos Ventures for their Series A funding. Senior consumers (aged 50 and over) tend to hold stronger purchasing power, which makes them a very attractive demographic to target. Needless to say, however, getting them to actively participate in online-shopping is a challenge, and it was the most important problem our team set out to solve. Our product needed design considerations such as larger fonts and UI components in order to accommodate senior customers who may suffer from deteriorating eye sight. We also found that many of our customers preferred to make phone calls to ask for assistance rather than using embedded online chatting solutions. Many of our customers also had a difficult time with making payments online, to which our team made significant efforts to build the payment experience as seamless as possible. As we can see, the market for seniors is a constantly growing market, but it is a difficult market to crack due to the need to properly understand the different characteristics of senior customers.

Finding the Product Market Fit is Hard

The single most important factor for any startup to succeed - finding the product market fit. Can I confidently say that Fumi found a product market fit (PMF)? Not yet. In fact, I don’t think Fumi even came close to finding PMF during my 4 months there. Objectively speaking, Fumi is currently an e-commerce platform with an emphasis on senior fashion, with no next-day delivery and no brand-name recognition. However, we were able to see some explosive increase in active users when launching events such as the “Senior Model Contest.” The platform definitely grew significantly due to efforts on all fronts over the last 4 months, but it is still unable to demonstrate signs of sustainable & organic growth. There might have been some negativity in what I just said - but most startups that exist out there have also not found PMF, remember that :)

Back Office Application is Important for E-Commerce

The importance of a solid back office application for an e-commerce company is often overlooked. Obviously, as back office applications typically are not client facing, it often receives less attention from engineers (especially if the engineering team behind the product is also responsible for building the back office application). However, from my experience at Fumi, having a well thought-out back office application is extremely important for the business to function properly. A back office application is used to track inventory, respond to customer service needs, track delivery, and configure the customer-facing app to meet certain business needs. All of the above are a critical part of any e-commerce platform business. If the back office application is not properly set up, then there can be a lot of overhead for the developers, as all other teams (merchandising, marketing, customer relations, etc) constantly look to the developers to solve issues. For example, a customer relations manager may have the need to change user data on the database. If a back office application is not well set-up to perform such a task, they have no option but to ask the developers to do it for them. Over time, this can decrease the working efficiency of the software developers, as they constantly have to deal with external matters instead of being able to focus on the product itself. These kinds of issues only become more apparent as the business scales.

What I learned from an engineering standpoint will be covered in part two.

Github Authentication with SSH

Thu, 31 Dec 2020 00:00:00 +0000

How to setup local machine to properly authenticate Github with SSH

This blog post is meant to serve as a kind note for my future self, inevitably struggling to properly setup Github SSH authentication. If there’s anyone else out there who happens to stumble upon this post while struggling with the same issue, hopefully this post will help you out. I will go over what SSH authentication is, why you might be interested in setting one up yourself (in the context of Github), and finally, how to set it up properly.

What is SSH ?

SSH stands for Secure Shell, and is a cryptographic network protocol for establishing trustworthy and secure network operations over an unsecured public network such as the internet. Explaining the details of how exactly SSH works behind the scenes is not within the scope of this blog post, so I’ll keep this explanation at a high-level overview. SSH Authentication is generally implemented via a classic client-server model, where the connection between the client and server is initiated by an SSH client making a connection request. The server, upon receiving a connection request from a client, sends a public key over to the SSH client, starting a process well known as public key cryptography (if you are not familiar with it, and would like to learn more about it in depth, feel free to check this link out). Once the SSH client and SSH server agree that each other can be trusted, the two establish a communication channel in which data is encrypted, making it secure.

Client-Server Model.

Why Use SSH for Github ?

There are two different ways to interact with remote github repositories (clone, pull, push, etc): HTTPS and SSH.

Github SSH Screenshot.

HTTPS is a very popular option for a lot of Github users, because it is extremely simple. In fact, Github also recommends HTTPS over SSH due to its port (443) being globally accessible, even behind firewalls. If you opt for HTTPS, all you need is your Github account information (id / password). However, when interacting with remote repositories, you will be prompted to enter your Github credentials every time (yes, even if you cache your credentials somewhere, it is still being sent over the network for authentication). This is where I see the comparative advantage of setting up SSH for authentication. Using HTTPS exposes your Github account credentials in some way, shape, or form. If your account information is compromised, you may be locked out of all your repositories and your account itself. That can be a real disaster.

On the other hand, if you opt for SSH, you do not have to enter your Github credentials at all. It is more secure in the sense that you do not expose your Github account. However, in order to set up SSH on your local machine, you do have to create a public/private key, add the private key to your ssh-agent, and copy the public key to your Github account. This way, Github can verify that it is indeed you that is trying to interact with Github. Let’s take a look at how to set all this up!

How to Set up SSH for Github

Generate a Key Pair

The first step in the process is to generate a public-private key pair. Use the ssh-keygen command to generate your keys.

$ ssh-keygen -t ed25519 -C "youremailaddress@example.com"

You will have been prompted for a location to store your SSH keys, and I’ll leave that up to you. As a note, the default location is ~/.ssh. The ssh-keygen command should also have prompted you to enter a passphrase. It’s up to you to add a passphrase or not, but for security purposes, I would highly recommend using one. Now, check if your keys have been properly generated:

$ cd ~/.ssh # or wherever you specified your keys to be saved
$ ls
id_ed25519      id_ed25519.pub

Now start the ssh-agent in the background:

$ eval "$(ssh-agent -s)"

Now it’s time to add the config file for SSH:

$ vim ~/.ssh/config
# add the following in the config file and save
Host *
  AddKeysToAgent yes # automatically load keys to agent
  UseKeychain yes # store passphrases in keychain
  IdentityFile ~/.ssh/id_ed25519 # or wherever you specified your keys to be saved

Finally, add your SSH private key to the ssh-agent and store your passphrase in the keychain.

$ ssh-add -K ~/.ssh/id_ed25519

That’s it! You now have successfully set up your local machine. But we still need to copy the public key that we’ve generated to Github. Run the following command to copy the content of your public key file on your clipboard:

$ pbcopy < ~/.ssh/id_rsa.pub

Now, head over to your Github account > Settings > SSH and GPG keys. Click on “New SSH Key”. Enter the title of your SSH key (try to be descriptive, just in case you add more SSH keys later on), and copy the public key content from your clipboard.

Screenshot of Github Add SSH.

You should see that your public key is now added to your Github account. You are all set now!

SSH Port Connection

After following all the steps from above, everything should be working fine. But as with everything else, there’s a caveat. Your SSH connection will default to port 22, which is the standard port number for SSH. However, unlike HTTPS, certain firewalls or networks may be blocking that port, in which case your SSH connection will hang. Fortunately, there is a workaround. We can simply configure SSH to establish connection on port 443, which is the default port for HTTPS.

$ vim ~/.ssh/config
# we already added this part before
Host *
  AddKeysToAgent yes # automatically load keys to agent
  UseKeychain yes # store passphrases in keychain
  IdentityFile ~/.ssh/id_ed25519 # or wherever you specified your keys to be saved

# TODO : set github ssh connection to port 443
Host github.com
  Hostname ssh.github.com
  Port 443

Now your SSH setup with Github should really be working!

Setting up ESLint and Prettier with VS Code

Sat, 08 Aug 2020 00:00:00 +0000

Setting up ESLint and Prettier with VS Code

An important part of software development/engineering (or whatever else you may want to call it) is writing code that works as intended, and is as bug-free as can be made. However, there’s actually a lot more to writing code than just to “make it work.” This is primarily because almost any meaningful project/software that exists in the world is created through collaboration. In other words, it is important that you are able to easily read others’ code, and that others are also able to easily read and understand your code. There are lots of ways to achieve this, including manual methods. A team of engineers can manually check for code style issues during code reviews, and possibly even create a detailed documentation on the coding styles that every engineer on the team should adhere to. However, with what we are about to cover in this blog post, investing engineers' time should no longer be put into reviewing code style. Without any further ado, let’s find out how to set up ESLint and Prettier with Visual Studio Code (VS Code).

Linter vs. Code Formatter

Before we dive into the actual set up, let us briefly discuss what a linter and a code formatter is.

What is a linter?

Lint, or a linter, is a tool that analyzes source code to flag programming errors, bugs, stylistic errors, and suspicious constructs. The term originates from a Unix utility that examined C language source code.

Linter is a developer tool that flags potential errors and bugs, which in turn lets the programmer easily spot potential bugs he/she may have accidentally introduced. For example, a linter can help catch some of the most commmon bugs such as import errors, undeclared variable errors, while also producing warnings where adequate; we all know about the tale of const vs let in JavaScript. While there are many linters to choose from, the most popular JavaScript/TypeScript linter in the market today is ESlint, so that’s what we will focus on in this blog post.

What is a code formatter?

A code formatter, unlike a linter, is only concerned with code style.

A code formatter will not be able to catch errors or bug in your code; that is not what it was built for. It is soley interested in beautifying your code according to configurable standards. There are many different code formatters available, but we will focus on a code formatter called Prettier, which is a highly popular and opinionated formatter.

Setting up ESLint and Prettier with VS Code

A linter and a code formatter are made to be used with a code editor. There are many code editors out there, the most popular ones being VS Code, Sublime Text, and Atom. VS Code, the most recent addition to the list, was made by Microsoft and has gained popularity over the last couple of years due to its rich collection of plugins that can all be configured directly within the application. In this blog post, we will specifically see how ESLint and Prettier can be configured for use with VS Code.

Understanding the Difficulty

The main reason why people find it tricky to properly set up ESLint and Prettier together is because ESLint does not strictly restrict itself within the boundaries of a linter; it also supports formatting code. There are style-related rules that can be configured within ESLint that may conflict with Prettier, which also wants to format your code. Hence, we need to make sure that we get avoid any possible conflicts between the two. With this in mind, let’s continue and embark on our journey of successfully configuring ESLint and Prettier to harmoniously work together.

Install ESLint and Prettier

Note: For official documentation and detailed guides, see official website for ESLint and Prettier.

We first need to install ESLint using npm or yarn, whichever one you prefer. You can either install ESLint globally, or locally. I personally use ESLint for all of my projects now, so I installed ESLint globally. Even if installed globally, we will still have to set up specific configuration files for each specific projects.

# install eslint globally
npm install -g eslint

Now that we’ve installed ESLint globally, we need to install Prettier. As with ESLint, Prettier can either be installed locally as a dependency, or can be installed globally. As I like to use Prettier with all my projects, I will install it globally.

# install prettier globally
npm install -g prettier

That’s it for basic installation! Now we have to configure ESLint and Prettier so that they can work together harmoniously within our editor.

Install ESLint & Prettier Plugin for VS Code

You will need to install the appropriate extension/plugin for your code editor. For VS Code, install ESLint and Prettier. These plugins have millions of downloads each, so it’ll be difficult to miss them in the VS Code extension marketplace. Once you’ve installed these, we are ready to move on to configuring each of them for proper usage.

Configuring ESLint and Prettier

As I mentioned earlier, ESLint and Prettier will conflict with each other if not set up with care. Fortunately, because this is such a widely recognized problem, there are packages that are made to solve exactly this problem. We are going to install two different packages; eslint-config-prettier to exclude all ESLint rules that could conflict with Prettier, and eslint-plugin-prettier to integrate the Prettier rules into ESLint rules. This way, you won’t have two different rules that conflict with each other.

npm install --save-dev eslint-config-prettier eslint-plugin-prettier

Note: Curious what the --save-dev flag does? By intalling with --save-dev flag, npm installs the packages eslint-config-prettier and eslint-plugin-prettier as devDependencies. This differs from the default behavior of npm install which installs it as a dependency to run the application, and places the downloaded files in node_modules.

Now that we have that successfully installed those packages, we need to run some commands to initialize ESLint within our project.

eslint --init
✔ How would you like to use ESLint? · style
✔ What type of modules does your project use? · esm
✔ Which framework does your project use? · none
✔ Does your project use TypeScript? · No / Yes
✔ Where does your code run? · browser
✔ How would you like to define a style for your project? · guide
✔ Which style guide do you want to follow? · airbnb
✔ What format do you want your config file to be in? · JSON
Checking peerDependencies of eslint-config-airbnb-base@latest
Local ESLint installation not found.
The config that you've selected requires the following dependencies:

eslint-config-airbnb-base@latest eslint@^5.16.0 || ^6.8.0 || ^7.2.0 eslint-plugin-import@^2.21.2
✔ Would you like to install them now with npm? · No / Yes
Installing eslint-config-airbnb-base@latest, eslint@^5.16.0 || ^6.8.0 || ^7.2.0, eslint-plugin-import@^2.21.2
npm WARN eslint-plugin-prettier@3.1.4 requires a peer of prettier@>=1.13.0 but none is installed. You must install peer dependencies yourself.
npm WARN eslint-prettier-blogpost@1.0.0 No repository field.

+ eslint-plugin-import@2.22.0
+ eslint-config-airbnb-base@14.2.0
+ eslint@7.6.0
added 171 packages from 100 contributors and audited 176 packages in 10.789s

24 packages are looking for funding
  run `npm fund` for details

found 0 vulnerabilities

As illustrated above, when you run the command eslint --init, you will be asked questions to determine the configurations of the linter. You just need to answer those questions based on the project you are working on. One thing to note here is the question “Which style guide do you want to follow?” There are many different style guides out there, including the JavaScript standard style guide, Airbnb style guide, and Google style guide. These are some very popular guides that a lot of developers choose to use (I chose the Airbnb style guide, which seems to be the most popular style guide out there). If you or your team have your own set of style guides, you don’t necessarily have to select an existing one.

Once the setup is successful, you should see a file eslintrc.{js/json/yaml} based on the format you chose your configuration file to be in. For Prettier to run with ESLint, your configuration file should look something like this:

{
  "env": {
    "browser": true,
    "es2020": true
  },
  "extends": ["airbnb-base", "prettier"],
  "plugins": ["prettier"],
  "parserOptions": {
    "ecmaVersion": 11,
    "sourceType": "module"
  },
  "rules": {
    "prettier/prettier": ["error"]
  }
}

Note: You can include files/directories that you want your linter to ignore in a file called .eslintignore (very similar concept to a .gitignore file).

Now, we just have to one final step. We need to set up Prettier locally, and tell Prettier how we want it to format our code.

# install prettier as local devDependency
npm install --save-dev prettier

That should have installed Prettier as a local devDependency. Now, we need to create a configuration file for Prettier, and tell it how to format our code.

# in your project root directory
touch .prettierrc

Open .prettierrc in your code editor, and set up a configuration for Prettier.

{
  "semi": true,
  "trailingComma": "all",
  "singleQuote": true,
  "printWidth": 70
}

This is simply a possible configuration, but feel free to take a look at other options here.

VS Code Configurations

So, you may still be asking yourself how we would actually use ESLint and Prettier. Will it work automagically? One thing we need to understand is that the way we set up Prettier to run as an ESLint rule. We have excluded all conflicting rules through eslint-config-prettier, and integrated Prettier as an ESLint rule through eslint-plugin-prettier. As a result, we only need to run ESLint to achieve the results we want for both linting and code formatting. ESLint can be run in mainly two different ways. We can use the ESLint CLI to manually run the linter (in the form of eslint index.js), or we could set up our VS Code configurations in such a way that will automatically format the code on save. To clarify, ESLint will check your code for possible bugs and errors even if you don’t manually run it via the CLI, but will not format your code automatically if VS Code configurations aren’t set up to do so. I like to have my code formatted automatically on save, so I will tweak my VS Code configurations to do exactly that.

// VS Code Settings JSON
// Set the default
"editor.formatOnSave": false,
// Enable per-language
"[javascript]": {
    "editor.formatOnSave": true
}

With this set up, you should now be able to see that your code is formatted automatically when you save the file.

Bonus

Error Lens is a very useful VS Code plugin that I love. By default, ESLint will underline error-prone code in red or orange, depending on how you’ve set up certain rules for ESLint. However, you will need to hover your mouse over the underlined code to see the error message.

A GIF of ESLint and Prettier working in VS Code without Error Lens.

You should be able to see that ESLint and Prettier are properly working! The red underline shows that a const variable should not be reassigned, and the code is automatically formatted when I save the file. However, notice that I had to hover over the red underline to actually see the error messages that ESLint has produced for us. With Error Lens, the error messages are printed on the screen, which in my opinion is very convenient. Let’s see that in action!

A GIF of ESLint and Prettier working in VS Code with Error Lens.

Voila! There we have it! ESLint and Prettier working beautifully with VS Code, and long gone are your worries of having to scratch your head over how to create a codebase with uniform code style. With Error Lens added to that combination, your day to day coding task should certainly be an experience! I hope this blog post helped you get started with using ESLint and Prettier for your JavaScript project, but please feel free to explore further by visiting their respective documentation.

Resize Images Stored in AWS S3 with AWS Lambda (feat. Docker)

Mon, 27 Apr 2020 00:00:00 +0000

A blog post on how to resize images stored in AWS S3 with AWS Lambda

Note: An AWS Account and a basic knowledge of their services are assumed throughout this blog post

I have been working on a team project for the past couple of months. The idea is to build a platform for off-campus room search, specifically geared towards students of Northwestern University. Django doing the heavy-lifting as the main engine to power the application, we were able to develop such a platform quite rapidly on our local development environment. We soon migrated our work to Heroku for production testing, and started serving media files from AWS S3 buckets. While user testing in preparation for a full launch of the application, a few non-trivial issues were found. As a room searching platform, our site needs to handle image requests adequately. However, our site suffered from a significant performance issue due to slow loading speed. Photos taken with the iPhones or the Samsung Galaxies these days typically are well over 2 MB in file size. Considering the fact that multiple photos are associated with a single room post and that multiple room posts need to be loaded on a single page, such an enormous file size quickly becomes prohibitive. We needed a solution to scale down large image files, without compromising image quality too much. Luckily, AWS already had advanced cloud computing services to help us achieve our goals: AWS Lambda.

1. What is AWS S3 & AWS Lambda?

The Amazon Simple Storage Service (Amazon S3) is a classic among a myriad of cloud services Amazon Web Services provide. S3 buckets simply provide space in remote Amazon servers for customers to store assets (or objects). It’s an attractive service because customers don’t have to worry about scalability, data availability, security, and performance themselves.

AWS Lambda is the other state-of-the-art technology we opted to use to solve our problem. Lambda lets you run code without provisioning or managing servers (hence, “serverless”), and you only need to pay for the total amount of compute time your application actually uses. All you need to do is upload your code and Lambda takes care of everything required to run and scale your code with high availability. Lambda is often configured to listen to certain event triggers (such as an object put event to an S3 bucket), which is exactly what we want to do. We want to have our Lambda function resize an image file, and store it back into an S3 bucket of our choice. Our goal is to significantly reduce loading time, by scaling images down to a more sensible size.

2. Create an Execution Role

We are not going to go through creating the buckets themselves. Creating the buckets are a rather trivial task - consult the documentation if necessary. Once the S3 buckets are set up, we will want to create a lambda function to attach to the buckets. Before we can create a lambda function, we need to create an execution role (AWSLambdaExecute) to grant explicit permission for our lambda function to manage objects in S3 buckets. Go ahead and create an execution role here, with the following properties:

Trusted entity – AWS Lambda
Permissions – AWSLambdaExecute
Role name - NameOfYourChoice

Once you have an execution role set up following the guidelines above, we can head over to the AWS Lambda management console to create the function.

3. Create the Lambda Function

Creating a lambda function is relatively straightforward. All we need to do is to navigate to the AWS Lambda management console, click “create function” and give it the execution role that we already created above.

A screenshot of AWS lambda create function page.

Once the function is created, you will be able to see the control panel for the lambda function that you have just created. Your lambda function doesn’t do anything yet, because we didn’t write any code. There’s an option to write code directly in the inline editor that AWS provides for us, but that’s only possible if your lambda function does not have any external dependencies. Unfortunately, for image resizing, an external library must be installed for pretty much any language you might be working in (in Python, our code needs Pillow to work properly). This is where it gets tricky. The lambda function executes in an amazonlinux environment, which most likely is not identical to your local environment. AWS provides a detailed guideline on how to solve this problem - you have to upload your function as a deployment package. In other words, you need to package your function as a module (with all dependencies included) and ship it onto AWS Lambda so that it knows exactly what it’s working with. The following is the lambda function that grabs an image from the source bucket, resizes it, then puts the resized image in the destination bucket.

# lambda_function.py
import boto3
from io import BytesIO
from os import path
from PIL import Image
   

s3 = boto3.resource('s3')
INCOMING_BUCKET     = 'incoming-bucket-name'
DESTINATION_BUCKET  = 'destination-bucket-name'
   

def lambda_handler(event, context):
    for key in event.get('Records'):
        object_key = key['s3']['object']['key']
        extension = path.splitext(object_key)[1].lower()

        # Grabs the source file
        obj = s3.Object(
            bucket_name=INCOMING_BUCKET,
            key=object_key,
        )
        obj_body = obj.get()['Body'].read()
  

        # Checking the extension and
        # Defining the buffer format
        if extension in ['.jpeg', '.jpg', '.png']:
            format = 'JPEG'

        # Image resize
        image = Image.open(BytesIO(obj_body))
        output_size = (1000, 1000)
        image.thumbnail(output_size) # image.thumbnail preserves aspect ratio, does does not exceed specified size, 
        buffer = BytesIO()
        image.save(buffer, format)
        buffer.seek(0)

        # Upload resized image to destination bucket
        obj = s3.Object(
            bucket_name=DESTINATION_BUCKET,
            key=f"{object_key}",
        )
        obj.put(Body=buffer)

        # Print to CloudWatch
        print('File saved at {}/{}'.format(
            DESTINATION_BUCKET,
            object_key,
        ))
        print("Image resize lambda handler completed.")

With the function written above, we can try to follow guidelines provided by Amazon to deploy our function as a package. Doing just that, I personally ran into issues with Pillow:

ImportError: "Unable to import module 'lambda_function': cannot import name '_imaging' from 'PIL'

It is difficult to say whether or not you will run into the same issue. For me, it happened, and it didn’t go away. After lots of googling, I came to a logical conclusion that it has to do with the amazonlinux environment that my lambda function is running in. The reason why I say a logical conclusion is because I did not get any errors when trying to reproduce the same error in the Python virtual environment that I packaged my function in:

A screenshot of the terminal showing import errors resulting from Python’s Pillow module.

After gaining confidence in where the general problem lies in, I narrowed down the issue: the problem seemed to be that the module PIL is already installed in the Python version that amazonlinux runs. PIL is shipped with Python by default, which is also the case in my local environment, except I can do:

$ pip uninstall PIL
$ pip install Pillow

This is in fact the recommended installation procedure outlined in the official documentation of Pillow. However, this is not possible with AWS Lambda, because I have no direct ownership over the machine that the lambda function runs on. Of course, if you happen to be running the lambda function on an AWS EC2 instance that you have purchased, you are in luck. If not, don’t worry. Docker to the rescue.

4. Docker to the Rescue

Docker’s logo.

Credits to freecodecamp tutorial

Docker technology is simply life-saving. I have no intensive background in working with Docker, but I was able to take a sneak peek at what Docker can offer to software developers. A simple introduction to Docker from their very own website: “Containers are a standardized unit of software that allows developers to isolate their app from its environment, solving the “it works on my machine” headache. For millions of developers today, Docker is the de facto standard to build and share containerized apps - from desktop, to the cloud.”

Let’s get started by spinning up a Docker container. This Docker container will be able to see the lambda function code that you have written.

$ brew cask install docker
$ cd /your/path/to/aws_lambda_function
$ docker run -v <directory your code is in>:/aws_lambda -it --rm ubuntu

The docker command above does the following:

The -v flag makes your code directory available inside the container in a directory called “aws_lambda”.
The -it flag means you get to interact with this container.
The --rm flag means Docker will remove the container when you’re finished.
ubuntu is the name of an official container, Ubuntu. If this container image isn’t already on your machine, Docker will download it for you.

You should now see something like this:

root@c1996f32a397:/#

We now have to do a couple of “Ubuntu” things, before we can start baking the package to deploy on AWS Lambda.

$ apt-get update
$ apt-get install python3-pip
$ apt-get install zip

That just got us everything we need to start installing our dependencies within the Docker container. What we are about to do is fairly simple, and is documented in the AWS documentation. The only difference is that we are doing it within a Docker container instead of your local machine.

$ cd aws_lambda 
$ ls # you should be able to see your function source code here 
lambda_function.py
$ pip3 install -t . Pillow # install package specifically into current directory
$ zip lambda_function.zip lambda_function.py # zip lambda_function.py 
$ zip -r lambda_function.zip PIL # zip downloaded package PIL 
$ zip -r lambda_function.zip Pillow* # zip downloaded packages Pillow-7.1.2.dist-info, Pillow.libs
$ exit

You should now see lambda_function.zip within your local directory that has your lambda_function.py file. The container is automagically removed, because we set an --rm flag when we ran the docker command. The lambda_function.zip archive is ready to be uploaded to AWS!

5. Deploy Your Package on AWS

The one final step remaining is to upload your zip file onto AWS. If your zip file is under 10 MB in size, you can upload it directly from the AWS Lambda console. If not, it is recommended that you upload it to an S3 bucket first, then pull in the S3 bucket URL from AWS Lambda console. Once your package is deployed, you should be able to see your lambda function at work whenever the event you set is triggered.

There’s just one last thing that I want to point out. In this blog post, we have been working with two different S3 buckets: grab from the source and put in the destination. I want to note that it’s possible to work with just one bucket. There’s a caveat there, however. Carefully think through what will happen if we configure our lambda function to trigger upon a put event at the S3 bucket. Yes, an infinte loop. Why? The initial put event will trigger the lambda function, which will resize the image, and put it back in the same bucket. Then, the lambda function will be triggered again. It will not stop. Amazon will happily serve your lambda function by scaling the resources needed for your lambda function to keep on running an infinite loop. But you certainly won’t be happy with that, or at least your wallet won’t be. To prevent such a disaster from happening, you must configure a prefix for your lambda function trigger. Instead of listening to all put events, it will now only listen for a specific put event that matches a certain prefix. Of course, you will also have to make sure that the lambda function uploads to a path that does not start with the prefix you have set.

Phewww. That was a lot! If you’ve followed along, you now should have a functional lambda function resizing images for you behind the scenes. To recap, here are the steps we took:

Create S3 bucket(s)
Create an execution role for AWS Lambda
Create an AWS Lambda function
Write code for your function
Package your function using Docker
Upload the packaged function on AWS
Enjoy!

Again, Docker really saved my life.

Django Channels 2.0, Postgres, Daphne, and Heroku

Mon, 20 Apr 2020 00:00:00 +0000

A blog post on how to deploy a Django-ASGI application on heroku

Disclaimer: This blog post is largely a quarantine coding production.

I already have a blog post on how to deploy a very simple Flask application on Heroku. In this blog post, I will explore more details of Heroku deployment, but this time with Django. I have been working on a Django & Channels side project, a web platform to enhance off-campus room searching experience for Northwestern students. As the application development was almost over and user testing seemed like the logical next step to take, I turned my eyes to Heroku (largely for their known simplicity compared to other services such as AWS or Linode). It turned out that there were a few curveballs thrown at me, largely due to Channels using an ASGI server rather than a traditional Python WSGI server. I hope this blog post serves as a future reference for myself, and for a few other friends out there that may be facing similar problems that I did.

1. What is Heroku?

Heroku is an extremely popular Platform as a Service (PaaS). Heroku is very attractive for developers who need to deploy something quickly, without having to worry about the complications of building server-side infrastructure. Heroku has a free-tier, but only provides very limited resources. For small personal projects, the free-tier should get the job done, but for bigger projects, you may want to consider upgrading your plan to handle more traffic.

2. Background

In this post, I will be writing specifically about how to deploy a Django Channels application on Heroku. For those of you who are unfamiliar with Django Channels, here’s a brief introduction: simply put, Channels gracefully extends Django’s boundaries beyond HTTP - to handle WebSockets, chat protocols, IoT protocols, and more. It’s built on a Python specification called ASGI (Asynchronous Server Gateway Interface), which makes the deployment process slightly different from a Python WSGI application. This post will not be a Channels tutorial. If you are interested in learning more about Channels, take a look at their official documentation. In moving forward, I will assume that you have a basic understanding of Python and Git version control, that you already have a working Django Channels application, and that you already have files such as routing.py and consumers.py already setup. This blog post will deal with deployment related content only.

3. Configuring Django Settings

Environment variables play a big role in the separation of development and production environments. Rather than hard-coding environment variables (which is bad practice in general), setting separate environment variables in the local machine and Heroku deployment server makes life a lot easier for us developers. For macOS users, environment variables can be set in .zshenv or .bash_profile depending on your macOS version. The way I achieved a separation of settings for development and production environment was by conditionally setting production environment variables in a conditional where DEBUG is False. For this to work, the DEBUG value must be set to True in local settings, and to False in production. I used an environment variable named DEBUG_VALUE to dynamically assign the appropriate DEBUG value. You will also have to add an environment variable SECRET_KEY, which should not be hard-coded for production. We will deal with setting Heroku environment variables later. I also used django_heroku, which is an awesome package that automatically configures your Django application to work on Heroku. The use of django_heroku is in fact endorsed by Heroku, as it provides many niceties, including the reading of DATABASE_URL, logging configuration, a Heroku CI–compatible TestRunner, and an automatic configuration of ‘staticfiles’ to “just work” (according to Heroku).

$ pip install django_heroku

Because a Django Channels application requires an ASGI server, we are going to need something different than a standard WSGI server such as Gunicorn. The Channels project maintains an official ASGI HTTP/WebSocket server, Daphne, which I will be using in my project. However, there are alternatives such as uvicorn, which is also known to be stable. Because of this extra layer of complication, there are a few different ways one can go about configuring appropriate web servers depending on needs. For example, you may choose to keep running standard HTTP requests through a WSGI server and use Daphne only for things WSGI cannot do, like HTTP long-polling and WebSockets. You may also opt to utilize Daphne for all HTTP and WebSocket traffic, in which case Daphne auto-negotiates between HTTP and Websocket. For my own project, I decided to use Daphne as the only server to handle all requests.

$ pip install daphne

settings.py

# settings.py
SECRET_KEY = os.environ.get('SECRET_KEY')

DEBUG = (os.environ.get('DEBUG_VALUE') == 'True')

ALLOWED_HOSTS = ["localhost", "yourappname.herokuapp.com"]

ASGI_APPLICATION = 'yourappname.routing.application'

# you need channels_layers configured as follows, and channels_redis installed 
CHANNEL_LAYERS = {
    'default': {
        'BACKEND': 'channels_redis.core.RedisChannelLayer',
        'CONFIG': {
            "hosts": [os.environ.get('REDIS_URL', 'redis://localhost:6379')],
        },
    },
}

# Add the following at the end of settings.py
# Settings below are used for deployment
# Note that AWS settings are not required, but was needed for my application
if not DEBUG:
    import django_heroku
    AWS_STORAGE_BUCKET_NAME = os.environ.get('AWS_STORAGE_BUCKET_NAME')
    AWS_ACCESS_KEY_ID = os.environ.get('AWS_ACCESS_KEY_ID')
    AWS_SECRET_ACCESS_KEY = os.environ.get('AWS_SECRET_ACCESS_KEY')
    AWS_S3_FILE_OVERWRITE = False
    DEFAULT_FILE_STORAGE = 'storages.backends.s3boto3.S3Boto3Storage'
    S3_URL = f"http://{AWS_STORAGE_BUCKET_NAME}.s3.amazonaws.com/"
    STATIC_ROOT = os.path.join(BASE_DIR, 'staticfiles')  

    django_heroku.settings(locals())

asgi.py

Your asgi.py file should be located in the same directory as your wsgi.py file.

$ touch asgi.py

# asgi.py
"""
ASGI entrypoint. Configures Django and then runs the application
defined in the ASGI_APPLICATION setting.
"""

import os
import django
from channels.routing import get_default_application

os.environ.setdefault("DJANGO_SETTINGS_MODULE", "yourappname.settings")
django.setup()
application = get_default_application()

We are ready to move on to configuring Heroku now!

4. Configuring Heroku

First, you will need to download the Heroku CLI. For more information, please take a look here. During this process, you should have been prompted to create a Heroku account (please make sure you do, you’ll need that!) This first step should have been relatively simple and straightforward. Once you download the Heroku CLI, you will have to login.

$ brew tap heroku/brew && brew install heroku # install Heroku CLI for macOS
...installation complete 
$ heroku login
heroku: Press any key to open up the browser to login or q to exit: (press any key!)
Logging in... done
Logged in as yourname@someemail.com
$ git init # you will want to add a .gitignore file before you add files to commit!
$ git add -A
$ git commit -m "heroku deployment preparation"
$ heroku create name_your_own_app
$ heroku open 
... you should be seeing a Heroku default screen on your browser
$ git push heroku <your_local_branch_name>:master # if you exclude your local branch name, it'll default to master
Enumerating objects: 9, done.
Counting objects: 100% (9/9), done.
Delta compression using up to 4 threads
Compressing objects: 100% (5/5), done.
Writing objects: 100% (5/5), 434 bytes | 434.00 KiB/s, done.
Total 5 (delta 4), reused 0 (delta 0)
remote: Compressing source files... done.
remote: Building source:
remote: 
remote: -----> Python app detected
remote: -----> Installing SQLite3
remote: Sqlite3 successfully installed.
remote: -----> Installing requirements with pip
... push failed

It seems like our push to heroku master just failed. We just need to configure a couple more files to get our application up and running on Heroku. Now let’s take a look at what other files we need to successfully deploy our Django application.

Procfile

To deploy any application on Heroku, you will need to add a Procfile (without ‘.txt’, ‘.py’ or any extensions) specifying a web server and your application name. Without the Procfile, Heroku will not be able to run a web server, which is needed to serve your application.

$ # this file should be in your project's root directory 
$ vim Procfile
> web: daphne <your_application_name>.asgi:application --port $PORT --bind 0.0.0.0 -v2

Runtime.txt

runtime.txt is not absolutely necessary, but is highly recommended if you are looking to specify the Python version you want your application to run on Heroku’s servers.

$ # this file should be in your project's root directory 
$ vim runtime.txt
> python-3.8.2

I would highly recommend that you configure your runtime.txt with a Python version of 3.8.1 or higher. I initially deployed my application with Python version 3.7.4, which had a strange problem with opening a new database connection for each request, which crashed my server (my plan only allows for a maximum of 20 database connections). I spent hours searching through Google, which did not get me anywhere. Upgrading the Python version fixed the issue for me.

Requirements.txt

Having a requirements.txt is necessary for Heroku to automatically setup their dynos (their Linux machines) for you. Your requirements.txt must have any and all dependencies that your application needs to run properly. It is important that you remember to update your requirements.txt file everytime you add an external dependency to your project.

$ # this file should be in your project's root directory 
$ pip freeze > requirements.txt

Addons

You may have been using an SQLite database for development on your local machine. For production, however, using an SQLite database is not recommended. I decided to use Postgres for my application, which is a popular choice. Django Channels also requires a Redis instance to act as the channel layer, so we’ll also need to add heroku-redis.

$ heroku addons:create heroku-postgresql
$ heroku addons:create heroku-redis

Environment Variables

Setting environment variables for Heroku is very simple. They support configuration through both the Heroku CLI and their website dashboard. I configured mine through the CLI.

$ heroku config:set DEBUG_VALUE="False" # yes, "False" as a string!
$ heroku config:set SECRET_KEY="your_secret_key_in_strings"

Useful Heroku CLI Commands

Here are a couple very useful Heroku CLI commands.

$ heroku login 
$ heroku restart # restarts the Heroku dyno
$ heroku create <your_app_name>
$ heroku open # opens up your heroku app
$ heroku open -a <your_app_name> # when you are working outside your git repo 
$ heroku logs # opens up server logs, very useful for debugging
$ heroku addons:create <package_name> 
$ heroku pg:info # postgres database info
$ heroku run bash # opens a bash shell to Heroku machine
$ heroku run <your command> # e.g. heroku run python manage.py migrate
$ heroku certs:auto:enable # allow heroku to automatically manage ssl cert
$ heroku ps:scale web=1:<tier_name> # scale web dyno

5. Deploy Your Application

Let’s try it again now!

$ git add -A
$ git commit -m "deploying to heroku"
$ git push heroku <your_branch_name>:master
Enumerating objects: 9, done.
Counting objects: 100% (9/9), done.
Delta compression using up to 4 threads
Compressing objects: 100% (5/5), done.
Writing objects: 100% (5/5), 434 bytes | 434.00 KiB/s, done.
Total 5 (delta 4), reused 0 (delta 0)
remote: Compressing source files... done.
remote: Building source:
remote: 
remote: -----> Python app detected
remote: -----> Installing SQLite3
remote: Sqlite3 successfully installed.
remote: -----> Installing requirements with pip
remote: -----> $ python nu_housing_app/manage.py collectstatic --noinput
remote:        168 static files copied to '/tmp/build_d5d77a4a08320db00239af29729591fe/nu_housing_app/staticfiles', 520 post-processed.
remote: 
remote: -----> Discovering process types
remote:        Procfile declares types -> web
remote: 
remote: -----> Compressing...
remote:        Done: 86.2M
remote: -----> Launching...
remote:        Released v10
remote:        https://yourappname.herokuapp.com/ deployed to Heroku
remote: 
remote: Verifying deploy... done.

Note: Your logs may look different from mine. As long as it doesn’t error, there shouldn’t be a problem.

Seems like our application was successfully deployed! We just have one final step left; we need to initialize our database.

$ heroku run bash
Running bash on ⬢ your-app-name... ⣷ connecting, run.3028 (Hobby)
# inside the directory that has manage.py file, run the following
~ $ python manage.py makemigrations
~ $ python manage.py migrate
... you should see your database initializing ... and we are done!

Congratulations! We have successfully deployed our Django Channels application on Heroku. If you run into any problems, Heroku has done a great job in documenting how to deploy your application on Heroku, so check it out here.

What is a Graph?

Sat, 28 Dec 2019 00:00:00 +0000

A blog post on the graph data structure in computer science

In this blog post, I will be going over the basics of the graph data structure, including its representation, implementation, and basic graph traversal algorithms. Graph is an important data structure, especially because of its practical applications that can range from representing friendships to maps of locational intelligence, computer networks and much more. Does that sound interesting? Let’s dive into learning the basics of graphs.

1. Introduction to Graphs

Even if you are unfamiliar with graphs, you most likely have already seen one. If you have seen or worked with trees before, you have already encountered a graph before. Let me try to explain graphs in relation to trees, a data structure that you are more likely to be familiar with, or at least more so than graphs. All trees are graphs, but not all graphs are trees. Like trees, graphs also have nodes (or vertices) and edges that connect the nodes, but are not bound to the restrictions that trees have. A tree is an acyclic and connected graph. A graph, however, is not limited to such restrictions; a graph can have cycles and can be disconnected to form a forest consisting of multiple components that are disconnected with another. Take a look at the following to get a rough idea of what a directed, acyclic graph looks like.

A more mathematically oriented definition of a graph would be the following: a graph is a data structure that represents a finite set of nodes (or vertices) and set of edges which connect a pair of vertices. Confused about the technical terms (connected, directed, acyclic, cyclic, and more…) that were just thrown at you? Don’t worry, that’s what we’ll explore in the next section.

2. Categorization of Graphs

For some of you, terms such as acyclic, cyclic, directed, undirected, connected, disconnected may sound unfamiliar. Such categorization of graphs, however, are essential in deepening your understanding of graphs, and are not too difficult to get a grasp of either. To help your understanding, take a look at the following:

A diagram representing different kinds of graphs.

As clearly depicted (on purpose) above, there are many different ways to categorize a graph. Here are some of the most important categorizations of a graph to be aware of:

A connected graph has at least one vertex and there must be a path between every pair of vertices (i.e. every vertex must be reachable by following only the edges). Otherwise, we call it a disconnected graph. (We won’t go into the details of strongly connected and weakly connected graphs)
A directed graph’s edges have directions (often represented by arrows) associated to them. Otherwise, we call it an undirected graph.
A weighted graph’s edges have associated weights or values. Otherwise, we call it an unweighted graph. (Weights are often used to denote a cost or distance associated with two vertices. Depending on what we are trying to represent with the graph, the weights can carry different meanings. For instance, if a graph represents a map, a weight between two vertices may denote the distance between two locations (which are represented as two distinct vertices) on the map)
A cyclic graph contains at least one graph cycle. Otherwise, we call it an acylic graph.

It is important to have a very clear idea on the basic categorization of graphs based on certain characteristics (as listed above). As much as I would like to provide more detailed analysis on graph and its related theory, it is impossible to cover the entirety of graph theory in this blog post (nor do I have the knowledge to do so). If interested for more information on graphs (or graph theory), I found this article quite helpful.

3. Representation of Graphs

To dive deeper into the discussion of graphs, we need to understand how graphs are represented without having to visually draw a graph out. There are largely two different ways to represent any graph: as an adjacency list or as an adjacency matrix.

Let us first take a look at the adjacency list representation of a graph. A common way of representing a graph as an adjacency list is to directly map vertices to indices of an array, and at each index of an array, represent the neighboring vertices and weights in the form of linked lists. Take a look at the following example to help your understanding:

A diagram that explains representing graphs through an adjacency list.

The diagram above shows how an undirected, weighted graph can be represented as an adjacency list. Note that using a linked list is not the only possible way to implement an adjacency list (we could use hash tables instead, and use vertices as keys and associated weights as values).

Now let us examine the adjacency matrix representation of a graph. An adjacency matrix representation differs from an adjacency list in that an adjacency matrix requires a |V| × |V| matrix (where |V| denotes the number of vertices) to store information on the existence of an edge between two vertices (if the graph is unweighted) or the weight between two vertices (if the graph is weighted). A non-existant edge is usually represented by a 0 for an unweighted graph, and by “inf” (infinity) for a weighted graph. To visualize the explanation, take a look at the following adjacent matrix representation of an undirected, weighted graph:

A diagram that explains representing graphs through an adjacency matrix.

The diagram above should give a clear idea of how to represent a graph with an adjacency matrix. If we were to represent an unweighted graph, the “inf"s would be replaced with 0s, and all the weights (represented in red numbers) would be replaced with 1s. Now, one might ask “Which representation is better?” Well, it depends. If optimizing for speed, an adjacency matrix representation will guarantee an O(1) lookup time when searching for an edge between two vertices. On the other hand, an adjacency list implemented with a linked list would give O(n) time, since in the worst case, we would have to search through the entire adjacency list of vertex a to find that there exists an edge between, say, a and f. If we were to optimize for space, however, an adjacency list would yield a significant advantage especially when representing a sparse graph. An adjacency list will only store information in a linked list if there exists an edge between two vertices. However, as explained above, an adjacency matrix always requires a |V| × |V| matrix. Hence, if a graph is sparse (i.e. there are very few edges between vertices), the adjacency matrix will be full of default values that indicate that there is no edge between two vertices, which is quite space-inefficient.

4. Traversing a Graph

Now that we have a solid idea of the graph data structure, it’s time to discuss how to do a graph walk (or a graph traversal ). Knowing how to traverse a graph is extremely important, because most advanced graph algorithms (such as Dijkstra’s Algorithm) use graph traversal algorithms as a fundamental cornerstone. There are largely two different approaches for traversing a graph; depth first and breadth first. A depth first traversal explores a graph until it is impossible to move further, backtracks, and continues exploring the graph in a depth-first manner until all nodes are visited. A breadth first traversal explores all the immediate neighbors first before moving on deeper into the graph, and continues exploring in a breadth-first manner until all nodes are visited. Both algorithms look quite similar, but a depth first traversal is usually implemented with a stack, whereas a breadth first traversal uses a queue. Let’s take a look at the following code that implements both algorithms.

import queue

def dfs(graph, start_node): 
    #this is an iterative implementation of a depth-first search
    #using an auxiliary stack data structure

    nodes = [start_node]
    visited = set()

    while len(nodes) > 0: 
        currnode = nodes.pop()
        if currnode in visited: continue
        visited.add(currnode) #mark currnode as visited
        for neighbor in graph[currnode]:
            #goes through the entire list of neighbors for currnode
            nodes.append(neighbor)

def bfs(graph, start_node): 
    #this is the breadth-first algorithm
    #uses a queue instead of a stack

    nodes = queue.Queue(maxsize=0) #queue with maxsize inf
    visited = set()

    while nodes.qsize() > 0: 
        currnode = nodes.get()
        if currnode in visited: continue
        visited.add(currnode) #mark currnode as visited
        for neighbor in graph[currnode]:
            #goes through the entire list of neighbors for currnode
            nodes.put(neighbor)

Notice how both the depth first traversal algorithm and the breadth first traversal algorithm are very similarly structured. In fact, the only difference is that the dfs function uses a stack (implemented with Python’s list) and the bfs function uses a queue from the Python’s queue module. It is also very important to keep track of the visited nodes, because graphs can have cycles that can lead to an infinite loop.

5. Conclusion

In this post, we went through the basics of the graph data structure. This post certainly doesn’t cover everything, but hopefully is enough to serve as the starting point in your journey to conquer the world of graphs. More advanced graph algorithms will likely be discussed in future posts.

Finding the Shortest Path from A to Anywhere

Sun, 15 Dec 2019 00:00:00 +0000

A deep dive into Dijkstra's Algorithm

In this post, I would like to take a slightly different approach and write about an algorithm that I learned about in a classroom setting. While taking a computer science course in data structures and algorithms this quarter, I found the graph data structure and related algorithms to be particularly fascinating, as graphs can be applied in real-life to represent virtually anything from friendships to geographic maps. Although far from being an expert on this topic, I am going to make an attempt to dissect perhaps the most famous graph algorithm and its practical application. Take a bow, introducing Dijkstra’s algorithm and the single source shortest path problem.

1. Single Source Shortest Path

Before we talk about the algorithm itself, let’s first get a sense of what Dijkstra’s algorithm is all about; what significant questions can Dijkstra’s famous algorithm answer? Often referred to as the “Single source shortest path” problem, Dijkstra’s algorithm is suitable for finding the shortest distance from a single vertex to all other vertices. Using the results Dijkstra’s algorithm produces, we can also find the shortest path from a single vertex to a specific destination, say, vertex f.

A diagram of a graph denoting the start node and target node for finding the shortest path between two nodes.

The graph illustrated above might have you think “It doesn’t seem that hard to manually find the shortest path from vertex a to f !” If that’s the case, imagine a graph with hundreds of vertices and edges; would you still be willing to manually find the shortest path from a to f ? It would be possible, but extremely time-consuming. This is where Dijkstra’s algorithm comes into play. With some help from Dijkstra, we can find the shortest path from a single vertex to every other reachable vertex in the graph, with relative ease.

2. Dijkstra’s Algorithm

Finally, we can start our dissection of Dijkstra’s algorithm. If we simply try to memorize the code for the algorithm line by line, it will be difficult (at least for people with non-photographic memory, like me). The better approach would be to try and understand what repeated steps the algorithm goes through until it finishes processing the graph and returns. The following are the steps that Dijkstra’s algorithm follows, until it exhausts the entire vertex set of a graph:

When selecting the next node to visit, decision should be made based on the known distance
For each node we visit, we check the neighboring nodes (i.e. the nodes that are 1 edge away from the node we are currently visiting)
When checking the neighboring nodes, calculate the distance traveled so far (distance from the starting node to the current node) + distance to the neighboring node from the current node
If the calculation done above is less than the known distance to the neighboring node, update the distance information for that neighboring node

(Note: Steps 3 - 4 are known as the relaxation of vertices)

In the implementation of the algorithm, I will be using Python’s heapq module, which conveniently lets us use a priority queue to ensure that the nodes are relaxed in the correct order (minimum distance nodes first!). Also, note that I will be using an adjacency list representation of graphs, implemented with Python’s dictionary. With the steps specified above in mind, let’s take a look at the following implementaion of Dijkstra’s algorithm:

#python implementation of Dijkstra's algorithm using priority queue 
import heapq 

def dijkstra(graph, source):
    dist = { vertex: float("inf") for vertex in graph }
    pred = { vertex: -1  for vertex in graph }
    visited = { vertex: False for vertex in graph }
  

    dist[source] = 0
    pred[source] = source
    todo = [(0, source)]
  

    #repeat the process until nodes are exhausted
    while len(todo) > 0:
    	#select the node with minimum known distance
        curr_distance, curr_vertex = heapq.heappop(todo)
      

        #if vertex has already been visited, skip it
        if visited[curr_vertex]: continue

        #if vertex has not been visited do the follwing
        visited[curr_vertex] = True
      

        #check the neighbors of the current node
        for neighbor, weight in graph[curr_vertex].items():
        	#update if following condition is satisfied
            if curr_distance + weight < dist[neighbor]:
                dist[neighbor] = curr_distance + weight
                #keep track of previous node : we need it to find shortest path
                pred[neighbor] = curr_vertex
                heapq.heappush(todo, (curr_distance + weight, neighbor))
    #result packages dist and pred in a tuple     

    return (pred, dist)

Dijkstra’s algorithm implemented as above does not directly give us the shortest path between two vertices. However, it does produce two valuable pieces of information, one of which can be carefully crafted to give us the shortest path between two vertices. Notice how the dijkstra function above returns a tuple containing pred and dist. The pred dictionary contains information on what node was visited before visiting each of the vertices while traversing the graph. The dist dictionary contains information on the distance from the starting point to each of the vertices in the graph. By working our way backwards in the pred dictionary, starting from the destination until we reach the starting point, it is possible to identify the shortest path from the source to destination. We will need the help of a stack to get the order correct.

def shortest_path(graph, source, destination): 
	#retrieve the pred dictionary from dijkstra
	pred = dijkstra(graph, source)[0]
	#we need a stack to reverse the order 
	stack = [destination]
	#we will return the shortest path as a list
	shortest_path = []

	while True:
		curr = stack[-1] 
		before = pred[curr]
		
		#source and destination are disconnected
		if before == -1: 
			return "there is no path!"
		
		#we have reached the source node by traversing backwards
		if before == source: 
			stack.append(before)
			break
		
		stack.append(before)

	#reverse the order: source --> destination 
	while stack: 
		shortest_path.append(stack.pop())

	return shortest_path

#example graph to test dijkstra's algorithm
example_graph = {
    'a': {'b': 6, 'c': 1},
    'b': {'c': 3, 'e': 9},
    'c': {'d': 10},
    'd': {'e': 2, 'f': 4, 'g': 13},
    'e': {'f': 2},
    'f': {'g': 7},
    'g': {}
}

With the shortest_path function, we can now find the shortest path from a source node to a destination node. Note that Dijkstra’s algorithm is not capable of finding the distance/shortest path between two nodes that are not connected.

A diagram that illustrates how the Dijkstra’s algorithm would proceed to find the shortest path between two nodes.

The following is the result of running dijkstra and shortest_path functions inside my Python shell with the example graph provided above. Note that the example_graph provided in the code is an adjacency list representation of the graph shown above, but is implemented with a Python dictionary.

{{ < figure src=“dijkstra_demo.png” caption=“A screenshot of my terminal showing the results of running Dijkstra’s algorithm.” >}}

3. Limitations of Dijkstra’s Algorithm

The Dijkstra’s algorithm is not bullet proof, however. The most remarkable limitation of Dijkstra’s algorithm is that it will not work as expected if there are negative edges in a graph. In order to find the shortest distance from one starting vertex to all connected vertices, Dijkstra relies on the fact that if all edges are non-negative, it is always true that adding an edge will never make a path shorter. Only when such an assumption can safely be made can Dijkstra’s “greedy” approach to the problem remain intact with solving global optimality. If negative edges are introduced and such an assumption is no longer valid, Dijkstra’s method of selecting a node with minimum known distance at each iteration will no longer guarantee a correct outcome. To overcome such limitations of Dijkstra’s algorithm, we can use other algorithms such as the Bellman-Ford’s algorithm and the Floyd-Warshall’s algorithm, as needed. These may be subject to discussion in future posts.

4. Conclusion

Hopefully this blog post helped you gain a solid understanding of Dijkstra’s algorithm and the single source shortest path problem. If there is anyone reading this post who is interested in learning the mathematical proof on the correctness of Dijkstra’s famous algorithm, I found this helpful.

Tweet Bot with Tweepy and Flask

Thu, 14 Nov 2019 00:00:00 +0000

How to build a Twitter bot with Python

According to recent research conducted by USC and Indiana University affiliates, the conservative estimate for the number of Twitter bots is somewhere around 15% of all active accounts, suggesting that automated account activity is in fact prominent across the Twitter ecosystem. In fact, your daily interactions on Twitter may very well include interactions with bots! Despite the negative connotations surrounding the term Twitter Bots, the qualitative judgement shouldn’t be so simplistic. The usage of Twitter bots can vary from being as helpful as disseminating information on natural disasters, to being as harmful as cultivating terrorist recruitment propaganda. Leaving that value judgment up to you, in this blog post, I’ll be writing about how to make a fully functional Twitter bot that Tweets simple sentences composed of three words - hence the “Three Word Philosopher.” Hop on!

1. Getting Ready - The Basics

As I always emphasize, when starting Python projects, it’s a good idea to default to creating a Python virtual environment for the specific project that you are working on. So let’s do just that:

$ mkdir my_twitter_bot
$ cd my_twitter_bot
$ python3 -m venv venv
$ source venv/bin/activate

Once we have our virtual environment ready, we are going to need to install some dependencies that we need for this project. For now, let us install the non-trivial dependencies that are must-haves to create a fully functioning Twitter bot. Make sure that you have your my_twitter_bot virtual environment activated!

$ #we are going to need BS4 for web scraping purposes
$ pip install beautifulsoup4
$ #we need tweepy to seamlessly interact with the Twitter API
$ pip install tweepy
$ #we need Flask to create a web app later on for user interface!
$ pip install Flask
$ #create requirements.txt containing all the dependency versions you've just installed
$ pip freeze > requirements.txt

We now have all the must-haves ready. Let’s move on to the next step. Again, recall the order of the steps we took to start out our project here: first create a virtual environment, then install any dependencies you are going to need for the project.

Note that in the snippet above, I’ve installed all major dependencies at once. However, in real life, this is unlikely to happen because you will never know which dependencies you are going to end up with at the end of the day. So in real life, it is more likely that you would update your requirements.txt file as you make progress with your project.

2. Feeding Your Bot - A Wealthful and Clean Text Corpus

To build a bot that doesn’t dully Tweet out same sets of hard-coded sentences over and over again, we need to first feed our bot with words so that it can dynamically generate sentences. With a bit of machine learning knowledge (such as the Markov Chain), it’s possible to generate a grammatically sound sentence given that the bot is fed with a wealthful repository of words. However, for this bot, I did not use any machine learning concepts and instead opted to try and mimic a short, “philosophical” sentence by performing a simple categorization of words. To achieve any of this, we are first going to need a source for our texts. Because I wanted a long text with “philosophical” words, I scraped an e-book of John Stuart Mill’s Utilitarianism from Project Gutenberg. You are more than welcome to choose any other source, but it’s going to be better if the text is decently long, to prevent repetitive sentences.

For the purposes of scraping an e-book (or any other source as long as it’s online), we are going to use BeautifulSoup, Python’s famous parsing library. BeautifulSoup is a fantastic package for parsing HTML and XML documents. We are not looking to do anyting complex here, so the code is fairly simple. Make a python file (I named mine scrape_words.py), and do:

#scrape_words.py
from bs4 import BeautifulSoup
import requests

URL = "http://www.gutenberg.org/files/11224/11224-h/11224-h.htm"

#use BeautifulSoup to extract content we want from an e-book from Project Gutenberg
def fetch_text(url):
    source = requests.get(url).text
    soup = BeautifulSoup(source, 'lxml')
    text_content = ''
    for el in soup.select('p'):
        text_content += el.text
    return text_content

NOTE: Web scraping isn’t illegal, but keep in mind that programs can send out requests at speeds that can have detrimental effects on the sites being scraped. It’s important to adhere to web scraping guidelines.

The few lines of Python code above are enough to extract the text content from the selected e-book’s HTML source. Since this is not a BeautifulSoup tutorial post, I will not go into the details of using BeautifulSoup as a web scraping tool for Python (A great, free tutorial can be found here, though). If you have been following along, you might have realized that the text content extracted from the HTML document tree still has some unwanted punctuations and sentences. We want to clean up that mess, or else our bot will be making sentences with unwanted words. In scrape_words.py add:

#scrape_words.py
import re
#text clean-up process
def text_cleanup(text):
    text = re.sub(r'\[.*?\]', '',text)
    text = re.sub(r'[^\w\d.\s]+', '', text)
    text = re.sub(r'\w*\d\w*', '', text)
    return text

output_file_write = open('gutenberg.txt', 'w')
output_file_write.write(text_cleanup(fetch_text(URL)))
output_file_write.close()

Essentially the function text_cleanup takes a string as input, gets rid of all the unwanted punctuations, and returns the cleaned up string. Then, we write that cleaned text into an output file, so our bot can later access it.

3. A Simple Categorization of the Corpus

Now that we have a clean text corpus ready for use, let’s write some code to help our bot generate sentences. Create a new file called sentence.py in your project directory, and do:

#sentence.py
import random

def categorize():
    f = open("gutenberg.txt","r")
    contents = f.read()
    listify = contents.split()
    body = []
    start = []
    end = []
    for word in listify:
        if word[0].isupper() and not word.isupper():
            start.append(word)
        elif word.endswith(".") and not word.isupper():
            end.append(word)
        elif not word.isupper():
            body.append(word)

    f.close()

    return [start, end, body]

The categorize() function reads the long text from the cleaned up gutenberg.txt file, and categorizes it into three different lists. The start list is the list of words that our bot is going to select from to put as the starting word of a three-word sentence. The end list is the list of words that our bot is going to finish the sentence with. The body list contains the remaining words that are neither the starting nor ending words. To choose the words to put in the start list, I leveraged the fact that proper nouns and words that mark the start of any sentence are capitalized. To choose the words to put in the end list, I leveraged the fact that words that mark the end of any sentence are immediately followed by a period. Anything that is not in the start or end list, I threw into the body list.

Although not necessarily the most sophisticated way of categorization, it will suffice for the purpose of feeding our bot with words to generate a sentence comprised of three parts - start, body, end.

4. Make That Bot Speak!

We’ve finished categorizing our words, so now we need to generate a sentence for the bot to Tweet out. We are going to accomplish that by using Python’s random module.

#sentence.py
def generate_sentence(start, end, rest):
    sentence = random.choice(start).replace('.', '') + ' ' \
    + random.choice(rest).lower().replace('.', '') + ' ' \
    + random.choice(end)
    return sentence

def run():
    return generate_sentence(categorize()[0], categorize()[1], categorize()[2])

That’s it! Now by calling the function run(), we can generate three-word sentences. Granted, because I didn’t use a more sophisticated algorithm (perhaps a machine learning algorithm) to try and emulate a more human-like sentence, the sentences generated will not always make perfect sense. However, for the purposes of this project, it does the minimum job of sounding like a “Three Word Philosopher.” Here is a sample of three-word sentences that our bot is capable of producing:

A screenshot of the terminal showing the kinds of sentences our bot is capable of producing.

5. Connecting our Bot to Twitter

We now have a bot that can generate three-word sentences (well, something close it) by selecting words from the three categorized lists we built. To build a bot that Tweets, we now need to plug our bot that resides in our local machine to the outer world - Twitter.

To connect our bot to Twitter, we first need to somehow programmatically access Twitter, right? That’s where Twitter’s API comes into play. API stands for Application Programming Interface (check this article out), and more specifically, Twitter’s API is a predefined set of Twitter-developed protocols that programmers can use to interact with Twitter at a programmatic level, without any user interface. Although Twitter’s API can be used by anyone who needs it, you do need to apply online to get access to your Twitter API keys (Twitter is very sensitive to malicious usage of their API). You will need to answer some questions to confirm with Twitter that you have legitimate reasons for requesting to use their API. Once Twitter approves, you will have access to your own API keys that you will need to connect your bot to Twitter.

Before we just copy and paste these API keys straight into our code, we need to first ask ourselves if it’s a good idea to hard-code configurations. My code is also pushed up to Github, which means that other people are going to be able to see it too. Uh-oh. Then it would be a good idea to encapsulate our API keys with a layer of protection, so other people don’t have access to sensitive information, right? Fortunately, Python’s dotenv module provides a fantastic way to get around such issues. After installing the dotenv package, you will need to create a .env file inside your project directory and store your sensitive information there.

$ pip install python-dotenv
$ pip freeze > requirements.txt
$ touch .env

Then open your .env file with an editor of your choice, and input the following:

#.env
CONSUMER_KEY='YOUR_CONSUMER_KEY'
CONSUMER_SECRET_KEY='YOUR_CONSUMER_SECRET_KEY'
ACCESS_KEY='YOUR_ACCESS_KEY'
ACCESS_SECRET_KEY='YOUR_SECRET_ACCESS_KEY'

If you are going to push your project up to a remote repository (i.e. Github), remember to include your .env file in your .gitignore file!

Good work, now we are ready to have our bot interact with Twitter API. There are numerous ways to achieve this using Python. In this project, I chose to use Tweepy, a very popular and easy-to-use Python library for accessing Twitter API. Create a file called tweet_generator.py and do:

#tweet_generator.py
from dotenv import load_dotenv
import tweepy, os

load_dotenv()

CONSUMER_KEY = os.getenv("CONSUMER_KEY")
CONSUMER_SECRET_KEY = os.getenv("CONSUMER_SECRET_KEY")
ACCESS_KEY = os.getenv("ACCESS_KEY")
ACCESS_SECRET_KEY = os.getenv("ACCESS_SECRET_KEY")

def tweet(status):
    auth = tweepy.OAuthHandler(CONSUMER_KEY, CONSUMER_SECRET_KEY)
    auth.set_access_token(ACCESS_KEY, ACCESS_SECRET_KEY)
    api = tweepy.API(auth)
    api.update_status(status)

    return status

The code above accomplishes the simple job of connecting our bot to Twitter through our authenticated API keys, and updating the status with a custom input - which we call Tweeting.

6. Taking it a Bit Further - Flask and Heroku

This is an optional part of the project that is not an integral part of the Twitter bot itself. However, I thought that it would be nice to have a user friendly interface for the “Three Word Philsopher” that I built so that people can easily use the bot that I built. I am not going to go into the details of Flask here, but as an introduction , Flask is a great Python microframework for building web applications. If you are interested, I have a more detailed post on Flask here. Create a Python file with a name of your choice, and do the following:

#flaskapp.py
from flask import Flask, render_template, request, redirect
#we need to import sentence and tweet_generator to access the functions we wrote!
import sentence, os, tweet_generator

app = Flask(__name__)

@app.route('/')
def main():
    #we need a HTML file called index.html to render it
    #the sentence parameter provides the template with dynamically generated sentences
    return render_template('index.html', sentence=sentence.run())

@app.route('/tweet', methods=['POST', 'GET'])
def tweet():
    #this function will handle the tweeting
    status = request.form['status']
    tweet_generator.tweet(status)
    return redirect('/')

if __name__ == '__main__':
    port = int(os.environ.get('PORT', 5000))
    app.run(host='0.0.0.0', port=port, debug=True)

You will need to create a HTML file index.html and put it in a directory called templates:

$ #inside your project directory
$ mkdir templates
$ touch index.html

Then open the index.html file in an editor of your preference:

<!DOCTYPE html>
<html>
<head>
    <title>Tweet Bot &#8211; Three Word Philosopher</title>
</head>
<body>
    <h5>
        <p>
            Philosopher's Tweet: "{{ sentence }}"
        </p>
    </h5>

    <div>
        <form action="{{ url_for('tweet') }}" method="post">
          <input type="hidden" name="status" value="{{ sentence }}">
          <div style="display:inline-block;">
            <button type="submit">
                Tweet
            </button>
          </div>
          <div style="display: inline-block">
            <a href="/" role="button">
                Refresh
            </a>
          </div>
          <div class="text-center" style="display: inline-block;">
            <a href="https://twitter.com/threethouthee" role="button">
                Visit the Philosopher
            </a>
          </div>
        </form>
    </div>
</body>
</html>

The HTML code above is not going to generate a decent looking interface. However, it does the bare minimum of displaying the tweets that the bot is generating through the “Refresh” button, and lets the user Tweet the generated sentence through the “Tweet” button. Again, if you are lost, you should take a look at this post to get a better understanding of Flask and its templating engine Jinja2. Check out my repository to see what I did to make my interface look a bit better.

Now that we have a Flask web application, I wanted to deploy it on Heroku so that I could reveal my bot to the world. Deploying a Flask Application on Heroku is not too difficult to do, and I have that covered in detail here, if you are interested.

7. What I learned

The Twitter bot project turned out to be an enjoyable learning experience. Here are some areas that I thought this project helped me hone my skills in:

Basic web scraping with Python
Python regular expressions to clean up the text corpus
Understanding the concept of APIs, and how to interact with them
Python configuration management best practices
How to deploy a Flask Application on Heroku

Possible improvements:

This bot would have been more interesting with a machine learning oriented algorithm to generate sentences that actually sound like real human, instead of three-word sentences.
Twitter API allows for much more than just Tweeting! It would be fun to implement other functionalities to the bot, such as automated Retweeting or automated replying.

That’s all! This is the link to the project site that I deployed on Heroku. Thank you for reading this post, and enjoy the philosophical Tweet bot!

By the way, I wasn’t kidding when I said there are bots lurking around in Twitter…

A real screenshot of another tweet bot retweeting a tweet that our bot just tweeted.

Github repository link.

Go Live With Your Flask Application on Heroku

Fri, 01 Nov 2019 00:00:00 +0000

How to deploy a Flask web application on Heroku

You worked for hours on your project, maybe days, or maybe even weeks or months. Regardless of the magnitude of your project, you probably want to share your work somewhere online so others can see it, right? At least for me, the urge to share my work online is something natural. An application is as good as dead if no one else can see your project, and is just sitting there on your local machine. Many inexperienced programmers go through that disheartening moment of thinking “How in the world do I host this application up on the internet?” If this sounds like you, this post is meant for you (because I know so very well how hopeless that feels), so follow along!

1. What is Heroku?

Heroku is an extremely popular Platform as a Service (PaaS). In case you are unfamiliar with what PaaS is, it’s a type of cloud computing services that allows developers to develop, manage, and run applications without having to deal with the complications of building server-side infrastructure.

2. Prerequisites

To deploy on Heroku, you will need a basic application to deploy (obviously). In this post, I will be writing specifically about how to deploy a Flask application on Heroku. Hopefully, your application is running in a virtual environment (it’s always a good idea, when working with Python). Additionally, a very basic knowledge of the command line and git would be of great help. Before we get started, if you don’t have an application, you can still follow along by cloning my repository to your local machine (you will need to have git installed):

$ git clone https://github.com/jeffjaehoyang/twitter_bot.git

You should see that you have a directory called “twitter_bot” now. Assuming you have pip and python3 installed:

$ cd twitter_bot
$ python3 -m venv venv
$ source venv/bin/activate
$ pip install -r requirements.txt

Note that the last command above installs into the virtual environment all the dependencies that the project needs to run properly. Once you run the commands above, we are ready to get started.

3. Starter

$ heroku login
  heroku: Press any key to open up the browser to login or q to exit: (press any key!)

That should have prompted you to a Heroku login page, where you need to type in your login credentials. Hopefully, you got that right. Now let’s move onto the specifics of deploying your Flask application on Heroku!

4. A Step Before Deployment

Install Gunicorn

To deploy your application on Heroku, you need to specify which web server your application is going to be running. We will use Gunicorn. We need to install this:

$ pip install gunicorn
$ pip freeze > requirements.txt

Note that we updated the requirements.txt above, because we added gunicorn as a dependency.

Add a Procfile

You will need to add a Procfile (don’t add any extensions!) specifying a web server and your application name:

$ vim Procfile
  web: gunicorn flaskrun: app

For those of you who are following along with your own application, please note that you should change flaskrun to the name of your application. We are ready to deploy!

5. Deploy Your Application

Inside your project directory, run:

$ heroku create YOURAPPNAME-api-heroku

Once you run the command above, you will be able to see two different URLs:

$ https://YOURAPPNAME-api-heroku.herokuapp.com | https://git.heroku.com/YOURAPPNAME-api-heroku.git

The first URL above is where your application is deployed, so please save it somewhere safe. The second URL is the Heroku git repository, and for your application to go live, you need to push your work to the master branch of the Heroku repository.

$ git push https://git.heroku.com/YOURAPPNAME-api-heroku.git master

6. Configuration on Heroku

If you were using my tweet bot repository to follow along, you may have realized that there’s a problem. Don’t worry, we didn’t do anything wrong. We just need to do one more thing. Because I used Twitter API keys for my tweet bot application (and that’s considered sensitive information), I didn’t push that information to my git repository. Locally, that information sits in a dotenv file, and on Heroku, we have to provide that information for our appliation to work. I won’t be providing my own API keys, but you can go ahead and look here for official documentation on how to get your own Twitter API keys. Once you have your own keys ready, configure them on Heroku by doing the following inside your project directory:

$ heroku config:set CONSUMER_KEY=YOUR-API-KEY
$ heroku config:set CONSUMER_SECRET_KEY=YOUR-SECRET-KEY
$ heroku config:set ACCESS_KEY=YOUR-ACCESS-KEY
$ heroku config:set ACCESS_SECRET_KEY=YOUR-ACCESS-SECRET-KEY

Congratulations! Your application should now be live on Heroku!

Heroku has done a great job in documenting how to deploy your application on Heroku: link.

Build Your Own Website with Flask

Sat, 26 Oct 2019 00:00:00 +0000

How to build your own website with Flask

Disclaimer: I have migrated my personal website since I wrote this post back in 2019, and moved away from the blog written with Flask. This website that you are on currently is built with Gatsby, and is hosted on Netlify. Regardless, for those of you interested in creating a simple personal website with Flask, read on!

Hello World! This is a post that was on my mind for a while now, and I’m so glad to finally write this post on this personal portfolio site I created! I previously had a Wordpress site that got me a good-looking fancy website up and running in no time. With all the beautiful Wordpress blog templates out there for a bargain, a sleek looking website was just a few clicks away. Nevertheless, something kept telling me to build a website myself - let me tell you why. Most importantly, the Wordpress site was beautiful on the outside, but rather bulky inside. I wanted to build a faster and a more light-weight website. Wordpress offered thousands of useful plugins (which is great), but having to update them constantly (for security purposes) was another downside of owning a Wordpress site. Last but certainly not the least, I thought “why not take this chance to brush up my web development skills?” This portfolio site is by no means greatly complex, but it still has all the basic structures of a full-fledged web application. So this simple site came to be, and for its creation, I mainly utilized Python’s microframework, Flask (and some BootStrap4 to make things look decent effortlessly).

1. What is Flask?

Flask is a lightweight microframework written in Python that is both intuitive and beginner friendly. I have a bit of prior experience in Flask development (mostly with APIs), but this was my first time building a Flask app completely from scratch, and I found the basics to be fairly easy to grasp. What I found was particularly convenient about Flask was Jinja2, a templating engine that comes with Flask (upon installation, literally). Jinja2 made it easy to create nested pages, by using a common layout page that all the nested pages could be built into. I decided to serve static Markdown files for my blog posts, instead of updating and editing directly from the website through a form. To achieve this, I used Flask’s FlatPages module, which essentially serves static Markdown files from a designated directory (all you need to do is upload a Markdown file to the directory, and it will be up on your website!)

2. Let’s Get Started!

Setting Up Your Virtual Environment

All developers who have any level of experience with Python will probably agree that no matter what type of project you are working on, installing packages systemwide is not a good idea. If you are new to Python, and don’t have a clue about what a virtual environment is, here’s an overly simplified explanation: Essentially, virtual environments help keep your dependencies (i.e. libraries/packages) separate for each project, so that each project knows exactly what versions of which extensions it needs to run properly. For more detailed information on Python’s virtual environment and best practices, I found this article particularly resourceful.

Create a virtual environment inside your project directory (assuming you are using Python 3):

$ mkdir my_flask_app
$ cd my_flask_app
$ touch run.py
$ python3 -m venv venv
$ source venv/bin/activate

A Quickstart to Your Basic Flask Application

Install the following dependencies using pip (assuming you have pip installed):

$ pip install Flask Flask-FlatPages Pygments

Now we have the bare minimum to start building a website like the one you are looking at right now! Inside run.py, write some code along the lines of the following:

from flask import Flask

app = Flask(__name__)

@app.route('/')
def home():
    return "Yes! Just Cooked Some Flask!"

#If running directly with Python, just set debug mode to TRUE
if __name__ == "__main__":
    app.run(debug=True)

Now we have the most basic application ready to go, so let’s run it:

$ python run.py
 * Debug mode: on
 * Running on http://127.0.0.1:5000/ (Press CTRL+C to quit)

For official documentation and more details on Flask, take a look here.

For those of you who need extra guidance, I highly recommend this Flask tutorial series. (It’s free!)

3. Serve the App, but How?

We have a very basic Flask application running, with the few lines of code we wrote above. If we are going to develop this baby Flask application into anything meaningful, we need to render pages on it (by pages, I mean blog posts, and not other pages; they can be written in pure HTML). There are a couple of ways to handle this task at hand. Something that would be a sensible thing to do for a dynamic website that needs to handle updates and deletes real-time would be to utilize Flask-SQLAlchemy, and use a relational database (such as SQLite). However, I was looking to do something a little bit different here. Since this website was meant to serve as a private online space, I figured that a relational database wasn’t necessary. Then I came across Flask-FlatPages, which made possible to serve static Markdown files to push onto the website. Setting up Flask-FlatPages shouldn’t be too difficult, let’s take a look.

from flask import Flask, url_for
from flask_flatpages import FlatPages

app = Flask(__name__)
app.config.from_object(__name__)
#Initialize flatpages
flatpages = FlatPages(app)

#Configure FLATPAGES settings
DEBUG = True
FLATPAGES_AUTO_RELOAD = DEBUG
FLATPAGES_EXTENSION = '.md'
FLATPAGES_ROOT = 'content'
POST_DIR = 'posts'

@app.route('/')
def home():
    return "Yes! Just Cooked Some Flask!"

#Configure URL routing for each blog post by the name of Markdown file
@app.route('/post/<name>')
def post(name):
    path = '{}/{}'.format(POST_DIR, name)
    post = flatpages.get_or_404(path)
    #Note that we don't have blog_post.html ready yet, so this won't work yet!
    return render_template('blog_post.html', post=post)

if __name__ == "__main__":
    app.run(debug=True)

The code above should get your application ready to serve your Markdown files. All you need to do is put your Markdown files inside ./content/posts:

$ cd content
$ cd posts
$ touch my_first_post.md
$ vim my_first_post.md

Inside my_first_post.md:

author: Jeff Yang
title: My First Blog Post
published: 2019-10-26

### This is a Heading
This is my **first** blog post.
> Let's Celebrate in a Blockquote!
`def this_is_python_code():
    print("you can even write code!")`

For further information on Markdown and its syntax, please check documentation.

Now we just need a HTML template to render this Markdown file into. As I mentioned above, Flask’s default templating engine Jinja2 is equipped with a simple but powerful “template inheritance” feature that makes life easy for us.

One level under my_flask_app directory, let’s create a directory called template and create layout.html and blog_post.html inside it:

$ mkdir templates
$ touch layout.html
$ touch blog_post.html

layout.html

<html>
<head>
    <title>My Flask App</title>
    <meta charset="utf-8">
    <meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
</head>

<body>
    <h1>
        <a href="{{ url_for('home') }}">Home</a>
    </h1>

    <div>
        {% block content %}{% endblock content %}
    </div>
</body>
</html>

blog_post.html

{% extends "layouts.html" %}
{% block content %}
    <h3><strong>{{ post.title }}</strong></h3>
    Published: {{ post.published }}
    {{ post.html|safe }}
{% endblock %}

The code for the HTML files above look a lot like pure HTML, but there are some oddities. Let’s take a look at some of the key features that the Jinja2 templating engine comes with.

{% extends "layouts.html" %}: blog_post.html is nested into layouts.html
{% block content %}{% endblock content %}: specifies the location of where your nested html file should be added to
{{ post.published }}: renders metadata from your Markdown file
{{ post.html|safe }}: renders the content of your Markdown file

Now your Markdown file content should be on your website, through the endpoint you configured before:

...
#Configure URL routing for each blog post by the name of Markdown file
@app.route('/post/<name>')
def post(name):
    path = '{}/{}'.format(POST_DIR, name)
    post = flatpages.get_or_404(path)
    #Now that we have blog_post.html ready to go, it should work!
    return render_template('blog_post.html', post=post)

For more detailed explanation on templating, please check the documentation for Jinja2.

4. Want to Insert Code Snippets for Blog Posts?

I knew from the moment I started building this website that I would need a way to insert code snippets inside a code block, with syntax highlighting. Using Wordpress, it was easy. I just had to install a plugin. Trying to add this feature to my own website, however, was a slightly different story. It’s extremely easy to find code snippets on the internet, they are on essentially every tech blog, we can see them on Stack Overflow, but it wasn’t as easy as I thought it would be to find exactly what I needed for this site. To save you from some trouble in case you are looking to add code snippets to your Flask application, here’s how to do it.

We already have most of the tools needed to insert beautiful and customized code blocks into the blog posts. Flask-FlatPages comes with Markdown, which comes with an extension called codehilite that supports syntax highlighting. However, we need to add some simple configurations to get it to work. We first need to install Pygments (but we already installed it in the very first pip install execution we did in the beginning of this article). If you didn’t install Pygments, go ahead and install it. Once Pygments is installed, we need to do the following:

from flask import Flask, url_for
from flask_flatpages import FlatPages

app = Flask(__name__)
app.config.from_object(__name__)
flatpages = FlatPages(app)

DEBUG = True
FLATPAGES_AUTO_RELOAD = DEBUG
FLATPAGES_EXTENSION = '.md'
FLATPAGES_ROOT = 'content'
#You need the extensions below for your code snippet to be properly displayed
FLATPAGES_MARKDOWN_EXTENSIONS = ['fenced_code', 'codehilite']

@app.route('/')
def home():
    return "Yes! Just Cooked Some Flask!"

#Configure URL routing for each blog post by the name of Markdown file
@app.route('/post/<name>')
def post(name):
    path = '{}/{}'.format(POST_DIR, name)
    post = flatpages.get_or_404(path)
    return render_template('blog_post.html', post=post)

To customize how your code block looks, do something like this:

$ pygmentize -L

The above code will print a list of built in themes available for your code blocks. To check out what the different themes look like (or customize the css based on your personal needs), run the following command inside my_flask_app directory:

$ mkdir static
$ cd static
$ pygmentize -S THEMENAMEOFYOURCHOICE -f html -a .codehilite > pygs.css

You should be able to see that a new pygs.css file is created inside the static directory. It should surely be working now! Ah, wait a minute, we forgot to link this CSS file to our HTML templates.

layouts.html

<!doctype html>
<html>
<head>
    <title>My Flask App</title>
    <meta charset="utf-8">
    <meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
    <link rel="stylesheet" type="text/css" href="{{ url_for('static', filename='pygs.css') }}">
</head>
...

That should do it. We finally have the backbones of a functional personal blog built with Flask! Let’s do a short recap of what we went through.

Created a virtual environment before we started to install anything needed to build this blog
Initialized a very basic Flask application
Created simple HTML templates powered by Jinja2
Went through how to serve static Markdown files through the HTML templates using Flask-Flatpages
How to insert prettified code blocks into our blog posts through some simple configurations of FlatPages-Markdown-Codehilite-Pygments

Congratulations! Now we are cookin’ with Flask!

Shipping Code for a Business – Is Your Application Scalable?

Mon, 09 Sep 2019 00:00:00 +0000

Reflecting on the 2019 summer internship at Knowru

Reflecting on 2019 Summer Internship @Knowru

This summer, I had the privilege of returning to and working at Knowru, a startup that primarily focuses on helping loan processing companies through a suite of innovative software. I had worked at Knowru in the summer of 2017 before enlisting in the Korean Army, and over the next 2 years of my military service, the company saw remarkable growth (most notably, the backing of a multimillion-dollar investment from Hong Kong). Back at Knowru as a software engineering intern, I had to navigate through problems that came from the company’s rapidly growing customer-base: writing code with scalability in mind.

Our team was tasked with the development of a web application that would allow loan processing companies access to foolproof verification of their customers (i.e. loan applicants). How? Our application would connect the loan agent and the customer through a video call, allow access (with agreement) to the GPS location of the customer, enable the loan agent to request certain photos, and provide agents a space to ask questions pertinent to determining the validity of a loan application. We built the prototype and sent it off to a Chinese micro-lending company for testing, and we waited—a few days later, we heard the ring of a message on Slack. (As we are software engineers working in office, it is often difficult to grasp the bigger picture, especially if there is a lack of knowledge on the business-side of the industry—therefore, such insight from testing is imperative). We were asked to re-design the application to support multiple companies and multiple loan products, as the application was only suitable for use by a single company.

As our lead frontend developer left the company soon after I joined, I became the main frontend engineer on the project. I discussed with the backend developer on how to best solve the problem. Our application was working like a charm—but only if there was a single company using the software. The code we wrote was simply not scalable, and from a business perspective, this was the prototype’s biggest flaw. Various companies would want varying fields for applicant’s personal information, request fields for different kinds of photos, and want their loan agents to ask different sets of questions to their customers. In other words, we needed to dynamically render the agent-side page of our web application so that it would suit each company’s wants and needs. I actively took advantage of the flexibility that React/Redux provided.

Through an API request, fetch JSON data containing specifications needed to dynamically render the page Store the specifications in the state of the application (Redux) For each React component, fetch necessary data needed and pass it as a prop Now we are fully equipped to render any page / component dynamically as long as we have the specifications provided by the company!

With the above, our application was now able to support any number of companies and loan products, just as long as we had the information provided by each company on what they wanted included in their own customized web application. The re-designing process was not too difficult or complicated, but I realized how important thinking out the design of the software is, especially when shipping code for business purposes. At the end, software engineers exist to solve complicated technical problems, and building a scalable product is integral for the final product to have a greater impact on the world.

Posts on Read Only Memory

My Story - Growth from IC3 to IC5

IC3 — Starting Out

IC3 → IC4 — Building Independence

IC4 — Broadening Scope

Switching to AI Data Infra

IC4 → IC5 — From Execution to Ownership

Looking Back

First 3 Months at Meta

Bootcamp Experience

Team Onboarding & Learnings

Configuring Portable Dotfiles

What is GNU Stow ?

How it works

1. Create a .dotfiles directory

2. Create subdirectories for all the programs you want to configure through dotfiles

3. Stow’em all!

Conclusion

Navigating a Computer Science Degree

Disclaimer

Computer Science is Not For Everyone

School Tips

Tip 1: Apply Yourself While at College

Tip 2: Utilize Free Online Resources

Tip 3: Don’t Cheat

Tip 4: Don’t Be Intimidated

Tip 5: Find a Mentor

Tip 6: Find a Hobby

Tip 7: It’s Okay to Take Time Off School

Tip 8: Start Learning About Money & Personal Finance Earlier

Tip 9: Explore Different Areas of Computer Science

Tip 10: Get Extremely Proficient with at Least One Programming Language

(Highly Opinionated) Tip 11: Learn How to Use Vim

Recruiting Tips

Tip 0: Know Your Data Structures & Algorithms (feat. Leetcode)

Tip 1: Think About Internships Early in College

Tip 2: Do Side Projects

Tip 3: Write a Proper Resume

Tip 4: Try to Contribute to Open Source Projects

Tip 5: Participate in Hackathons

Tip 6: Take Advantage of Networking Events on Campus

Tip 7: Don’t Compare Yourself to Your Peers

Tip 8: Getting Ghosted is Okay

A Practical Application of Redis Cache

1. What is Redis?

2. Installing Redis

3. How can Redis Help?

3. Setting up Redis Within an Express Application

4. Measuring Performance with Redis Implementation

Wrapping Up

My Facebook Internship

Application Process

Team Selection

A Remote Internship

Imposter Syndrome

Ramping Up

Not Just an Intern

Takeaways

Tips for Future Facebook Interns

Working at an Early Stage Startup (Pt. 2)

What did I work on?

Struggle 1 : Development Speed vs Quality of Code

Struggle 2 : Lack of Mentorship

Struggle 3 : Getting Used to Bad Engineering Practices

Struggle 4 : Communication

Positive 1 : Shaping Engineering Culture

Positive 2 : Experience Working Across the Stack

Positive 3 : Feel Like a Core Part of the Team

What’s next for me?

Working at an Early Stage Startup (Pt. 1)

Business Side of Things

Competition is Fierce in the Business of E-commerce

Market for Seniors is Big but Difficult

Finding the Product Market Fit is Hard

Back Office Application is Important for E-Commerce

Github Authentication with SSH

What is SSH ?

Why Use SSH for Github ?

How to Set up SSH for Github

Generate a Key Pair

1. Create a `.dotfiles` directory