Offload your API

​

Introduction

Most modern web applications provide rich product integrations that require fetching large amounts of data from third-party APIs.

For example, enabling users to sync their Hubspot contacts with your web application.

While calling the Hubspot API directly from your API’s controller would work, it might not be for all use cases.

Eventually, some users will start importing large sets of contacts, leading to:

• poor user experience (your user is waiting for your API response, without any progress indication)
• global degradation of your API performance (one of your API’s worker process will be unavailable during the whole import)
• failure to complete the import (the request to your API did timeout)

Moving the Hubspot API calls in a background function would provide a better user-experience:

• resilient product integration: an unavailable third party API won’t impact your API, the background function will retry in case of rate limiting
• continous feedback to the users: inform the user of the progress and status of the import

This example will show you how to implement and run a background function to batch import contacts from the Hubspot API from a web-app built with Express.

​

The importContactsFromHubspot() background function

As stated in the “Getting started” guide, we create our importContactsFromHubspot() function in the defer/ folder:

- src/
  - defer/
    - importContactsFromHubspot.ts ⬅
  - api.ts
  - hubspotClient.ts
- package.json
- .env
- ...

Here is our importContactsFromHubspot() implementation (inspired from Hubspot official Node.js tutorial):

import { hubspotClient } from "../hubspotClient";

const importContactsFromHubspot = async (userId: string) => {
  // ... get hubspot access token from user with Prisma
  const { hubspotAccessToken } = await prisma.user.findUnique({
    where: {
      id: userId,
    },
  });

  hubspotClient.setAccessToken(hubspotAccessToken);
  const getAllContacts = async (offset, startingContacts) => {
    // `console.log()` can be seen from Defer's dashboard
    console.log(`fetching contacts from offset: ${offset}`);

    const pageOfContacts = await hubspotClient.crm.contacts.basicApi.getPage(
      100,
      offset,
    );

    const endingContacts = startingContacts.concat(pageOfContacts.body.results);

    if (pageOfContacts.body.paging) {
      return await getAllContacts(
        pageOfContacts.body.paging.next.after,
        endingContacts,
      );
    } else {
      return endingContacts;
    }
  };
  const contacts = await getAllContacts(0, []);

  console.log(`fetched ${contacts.length} contacts`);

  // ...inserts contacts in DB...

  return contacts.length;
};

The importContactsFromHubspot() function fetches all contacts sequentially by batches of 100 contacts.

Once implemented, we wrap importContactsFromHubspot() with defer() and export it as default:

import { defer } from "@defer/client";
import { hubspotClient } from "../hubspotClient";

const importContactsFromHubspot = async (userId: string) => {
  // ...hubspot contacts API logic...
};

export default defer(importContactsFromHubspot, { retry: 5 });

Moved in a background function, the Hubspot APIs call will not affect your application’s performance, and will be retried by Defer in case of failure. Offering a resilient and responsive user experience.

The { retry: 5 } is a shorthand for up to 5 exponential back-off retries, as showcased below:

Our background function is now ready to be called from the API!

​

Calling importContactsFromHubspot() from the API

Let’s now call our importContactsFromHubspot() background function from our API.

import express from "express";
import importContacts from "./defer/importContacts";

const app = express();

app.post("/hubspot/import", async function (req, res) {
  const userId = req.session.userId;

  await importContacts(userId);

  res.send({ ok: true });
});

app.listen(3000);

That’s it 🎉

Your application now offers an improved Hubspot contacts integration:

• great user experience: the import is performed in the background without impacting your application’s overall performance.
• resilient import: the import will work for any number of contacts and will resume in case of Hubspot API rate limiting or outage.

This example will be soon updated to showcase how to integrate the progress of a background function in your product to keep the users in the loop (ex: display the status of the Hubspot contacts import in your application).