Recap

We ended the last session by getting our weekly cron tests passing. This session is about getting the biweekly tests passing.

How to schedule bi-weekly cron jobs

To get our tests passing our jobs will need to run every two weeks, and they'll need to do this even though the app is stopped and restarted, so far the cron time clocks in. And if for some reason the job couldn't run at the specified time we want it to retry.

But I very quickly got stuck on how to configure a bi-weekly schedule and posted a question on stackoverflow hoping to get some good ideas

Unfortunately, based on the friends I reached out to not having simple answers to this, and the fact that my stackoverflow question is still unanswered days after, it seems like this isn't a trivial problem or simply isn't possible with the cron specification, so it looks like I'm going to tradeoff having a fully automated bi-weekly cron job just so I don't get stuck in the weeds on a problem that doesn't really help achieve my project goal.

Last ditch research

Just making sure there isn't something obvious I missed, I check through @nestjs/schedule's convenience enum CronExpressions which contains regularly used patterns, but bi-weekly isn't on there.

I also find another nest-schedule module with a bit more functionality like the ability to specify max retries.

But after a time-box of about 30 minutes expires and no new information, I'm ready to move on.

The plan to get our bi-weekly tests passing

I have two main options I'm considering now:

A - generate the specific dates on which the biweekly job should run for at least a year in advance. Tech debt here being the need to continuously generate new dates or come back to fully automate this process.

B - Just run it every week, but don't run if the data we want to scrape is already in the db. Tech debt here being the work further down the pipeline needed to check our db for the info we need before our tests pass.

Given plan B's tech debt relies on work that was already going to come in I'm going with B.

Getting what tests we can passing for now

A few moments later we have one failing test which is going to keep failing till our tech debt is covered.

    test.skip('a week after Feb 19 at 4:30PM, scrapeContestData to NOT BE called', async () => {
      const service = app.get(TasksService);
      clock = sinon.useFakeTimers({
        now: new Date('2022-02-26T16:30Z').valueOf(),
      });
      await app.init();
      clock.tick(3000);
      expect(new Date().getDay()).toBe(6);
      expect(service._called).toBe(0);
    });

We also have two or more tests that don't really test what they should be testing. Here's an example of one:

    test('one minute after, TO HAVE BEEN called', async () => {
      const service = app.get(TasksService);
      expect(service._called).toBe(0);
      clock = sinon.useFakeTimers({
        now: new Date('2022-02-19T16:30Z').valueOf(),
      });
      await app.init();
      clock.tick(60000);
      expect(new Date().getDay()).toBe(6);

      expect(service._called).toBeGreaterThan(0);
    });

With this test, the intention really is... say our app was down and it restarted one minute after our cron time clocked in, we still want our job to 'run,' then transition to a 'have run' state, which we can check against.

However, instead of initializing our app one minute after the cron time to simulate a restart after the cron-time has passed, we are initializing it exactly at the cron time, so we're not actually testing the case we want to test here. I'm going to go ahead and make these and similar tests fail pending when the tech debt to get them passing is covered.

With that done, we have all the tests we can get passing for now all passing. And that pretty much wraps it up for this session.

We're moving onto the next session with 4 failing tests which we'll need to fix.


--------------------|---------|----------|---------|---------|-------------------
File                | % Stmts | % Branch | % Funcs | % Lines | Uncovered Line #s 
--------------------|---------|----------|---------|---------|-------------------
All files           |   88.37 |      100 |    87.5 |   84.84 |                   
 src                |   80.76 |      100 |      75 |      75 |                   
  app.controller.ts |     100 |      100 |     100 |     100 |                   
  app.module.ts     |     100 |      100 |     100 |     100 |                   
  app.service.ts    |     100 |      100 |     100 |     100 |                   
  main.ts           |       0 |      100 |       0 |       0 | 1-8               
 src/tasks          |     100 |      100 |     100 |     100 |                   
  tasks.module.ts   |     100 |      100 |     100 |     100 |                   
  tasks.service.ts  |     100 |      100 |     100 |     100 |                   
--------------------|---------|----------|---------|---------|-------------------
Test Suites: 1 failed, 1 passed, 2 total
Tests:       4 failed, 12 passed, 16 total
Snapshots:   0 total
Time:        7.359 s

Some refactoring and a pull request

I mentioned in the previous session that our tests could do with a for loop, since they all share the same structure, so I'm going ahead to do that quickly.

A couple of moments later and now we have just the one test, and an array of different cases. Using Jest's test.each method came in handy for this.

test.each(testCases)(
      '$name',
      async ({ dateString, clockTick, dayExpected, callsExpected }) => {
        const service = app.get(TasksService);
        expect(service._called).toBe(0);
        clock = sinon.useFakeTimers({
          now: new Date(dateString).valueOf(),
        });
        await app.init();
        clock.tick(clockTick);
        expect(new Date().getDay()).toBe(dayExpected);
        expect(service._called).toBe(callsExpected);
      },
    );

Our first PR

With that done I'm making our first PR to our main branch. Prior to this I've simply pushed directly to main but I want to come back to have a look at this code to see if there's anything obvious that needs changing before I merge it in.

Our next session

In the meantime I'm going to re-evaluate my decision to use PostgresDB over MongoDB for what will essentially be mostly json data.

Attributions

Cover Image: Thanks to Chuck Groom for surfacing it, and Vincent Dénie for creating it.

dozie.dev