Intro
I ended part one by saying:
In terms of optimizations, it'll be much faster to scrape contest data and cache it for O(1) look ups. If I do create a UI this is the approach I'll take. Of course that'll require jobs that periodically scrape the leetcode site each time a contest is out. Unfortunately this added complexity is the reason I've shelved putting up a UI.
In part II I explore implementing this optimization by building out a UI starting out with a rough architecture diagram
I decided to build this using a Postgres DB, NestJs for the backend, the React Library for the front-end, and to host it on AWS. At the time of writing this, I'll be biasing for an MVC approach if that's possible with NestJs, otherwise I'll simply host my React based front-end on vercel, probably building it with Create React App or NextJs.
Creating a cron job to scrape Leetcode contest data
First session - Getting familiar with Nest
It's my first time using Nest so there's going to be a bit of learning curve.
Looked at Nest's documentation to get started: https://docs.nestjs.com/
Stopped when I spun up a working hello world app
Second session - Pulling in a cron module
Skipped to the scheduling section in Nest's documentation, because the first thing I want to build is a cron job that regularly scrapes the Leetcode contest data on a weekly basis.
Again, the nest documentation was helpful. Skimmed through this info, then had a look at their example code
tasks.service.ts
@Injectable()
export class TasksService {
private readonly logger = new Logger(TasksService.name);
@Cron('45 * * * * *')
handleCron() {
this.logger.debug('Called when the second is 45');
}
@Interval(10000)
handleInterval() {
this.logger.debug('Called every 10 seconds');
}
@Timeout(5000)
handleTimeout() {
this.logger.debug('Called once after 5 seconds');
}
}
With NestJs it seems the idea is that we have modules, modules have controllers and controllers depend on services. Therefore services are like the workhorses where most of our implementation details will live, and controllers plumb services together? We'll see.
I stopped when I got the example code working locally
Third session will be writing unit tests for the scheduler I want to build
Third Session - Sketching out unit tests
To get started with the unit tests I start by sketching out an idea of what I want to achieve
tasks.controller.spec.ts
describe('scrapeContestData', () => {
describe('weekly', () => {
test.todo('Feb 13 at 4:30AM, scrapeContestData TO BE called');
test.todo(
'a day after Feb 13 at 4:30AM, scrapeContestData to NOT BE called ',
);
test.todo('a week after Feb 13 at 4:30AM, scrapeContestData TO BE called');
test.todo(
'2 weeks after Feb 13 at 4:30AM, scrapeContestData TO BE called ',
);
test.todo('one minute before ,should to NOT BE be called');
test.todo('one minute after, TO BE called');
test.todo('one hour before ,should to NOT BE be called');
test.todo('one hour after, TO BE called');
test.todo('one year later, TO BE called');
});
describe('biweekly', () => {
test.todo('Feb 19 at 4:30 PM, scrapeContestData TO BE called');
test.todo(
'a day after Feb 19 at 4:30PM, scrapeContestData to NOT BE called ',
);
test.todo('a week after Feb 19 at 4:30PM, scrapeContestData TO BE called');
test.todo(
'2 weeks after Feb 19 at 4:30PM, scrapeContestData TO BE called ',
);
test.todo('one minute before ,should to NOT BE be called');
test.todo('one minute after, TO BE called');
test.todo('one hour before ,should to NOT BE be called');
test.todo('one hour after, TO BE called');
});
});
The idea being that when I invoke a certain function or class method, it will either call scrapeContestData or not, depending on what the time is.
At this point I'm not entirely sure this is the correct way to think around testing the cron module I added in session 2, but it's a start.
So let's see how to flesh out these tests by thinking about it a bit.
From session 2 we know that the code which runs the cron jobs is bootstrapped in main.ts, and when the cron time clocks in, the log statement is printed, therefore any function we want to run should live within the cron handlers in our TasksService class.
Great! This makes it clearer that to check if functions in our cron handlers run. We simply need to mock TasksService and mock the date as well, run it (how?), then run an expect to have been called on the functions in our cron handlers.
Here's our TasksService class incorporating the idea of a scrapeContestData method:
tasks.service.ts
@Injectable()
export class TasksService {
private readonly logger = new Logger(TasksService.name);
scrapeContestData(): void {}
@Cron('45 * * * * *')
handleCron() {
this.scrapeContestData();
this.logger.debug('Called when the second is 45');
}
@Interval(10000)
handleInterval() {
this.scrapeContestData();
this.logger.debug('Called every 10 seconds');
}
@Timeout(5000)
handleTimeout() {
this.scrapeContestData();
this.logger.debug('Called once after 5 seconds');
}
}
If I were strictly following TDD I shouldn't be updating this class just yet, but oh well!
Now let's pop over to the tests to see if we can mock this class and implement at least some of our tests. I'll be relying heavily on this example test code to get my bearings:
app.controller.spec.ts
import { Test, TestingModule } from '@nestjs/testing';
import { AppController } from './app.controller';
import { AppService } from './app.service';
describe('AppController', () => {
let appController: AppController;
beforeEach(async () => {
const app: TestingModule = await Test.createTestingModule({
controllers: [AppController],
providers: [AppService],
}).compile();
appController = app.get<AppController>(AppController);
});
describe('root', () => {
it('should return "Hello World!"', () => {
expect(appController.getHello()).toBe('Hello World!');
});
});
});
A couple of minutes later....
We arrive at a reasonable looking test structure and our first failing test 🥳 🥳
tasks.controller.spec.ts
import { Test, TestingModule } from '@nestjs/testing';
import { TasksService } from './tasks.service';
const TasksServiceMock = {
scrapeContestData: jest.fn().mockImplementation(() => {
return;
}),
};
describe('TasksService', () => {
let tasksService: TasksService;
beforeEach(async () => {
const moduleRef: TestingModule = await Test.createTestingModule({
providers: [
{
provide: TasksService,
useValue: TasksServiceMock,
},
],
}).compile();
tasksService = moduleRef.get<TasksService>(TasksService);
});
describe('weekly', () => {
test('Feb 13 at 4:30AM, scrapeContestData TO BE called', () => {
expect(tasksService.scrapeContestData).toHaveBeenCalled();
});
...
Writing this test made me realize that for our TasksService class, we can directly name our cron handler scrapeContestData. So in our next session I'll focus on fleshing out all the tests and refactoring our TasksServiceClass in a TDD approach.
Fourth Session - Getting our cron tests passing
Part 1: Getting our Cron Tests Passing in NestJs 🎉
Part 2: Taking on Tech Debt to Handle Failing Bi-Weekly NestJs Cron Tests🎉
Part 3: Making use of Mocking and Dependency Injection to get NestJs Tests Passing 🎉
Project Discontinued
I decided to discontinue this project for two reasons. The first is that the time box I gave myself to complete this (end of March has elapsed) and the second is that I discovered Leetcode's virtual contest feature, which is essentially the feature this project set out to build.
Image links to Leetcode Journey - How to solve 1600+ LeetCode questions in one year?
Once I realised this pre-existing feature achieved my needs and given I wasn't planning to improve on it in anyway, I couldn't justify continuing this project.
However, I hope the the articles so far are helpful for people looking to work on a NestJs project, and that the Leetcode data scraping endpointthat I put up is useful to other people.
