How Frequently Our Customer Success Application Syncs Your Data

In a previous post, we looked at How We Built More Than 100 Customer Success-Integrations between Akita and apps like Hubspot, Salesforce, Intercom and Zendesk. We outlined our 3-step process for keeping data in sync: Schedule, Retrieve, and Process.

The Scheduler kicks everything off and is responsible for determining which data needs retrieving at any given time.

Note: Now might be a good time to re-read our previous post on How We Built More Than 100 Customer Success-Integrations for our Product!

Let’s look at a customer (ACME) who has connected their Salesforce account. We can sync many types of data (or Interaction Types) from Salesforce including:

  • Accounts
  • Contacts
  • Opportunities
  • Tasks
  • Notes
  • Users

By default, we sync each type of Salesforce data every 180 minutes. This may be overkill for “Users” (ie. new Salesforce licenses you might rarely add) but is useful for having timely access to new Accounts and Contacts in Akita.

We schedule the retrieval of each different Interaction Type for ACME (an Integration Object) separately. This simplified SQL statement allows us to select the Integration Objects that need syncing:

SELECT * FROM IntegrationObjects WHERE SyncedAt <= NOW() - INTERVAL ‘3 HOURS’;

The Scheduler runs every couple of minutes, finding any Integration Objects (ex. ACME’s > Salesforce > Accounts) that need syncing, creating a Retrieve Job and dispatching it to a queue. When the Retriever successfully executes the Retrieve Job, it will update the SyncedAt property to the current time.

The Retriever doesn’t re-sync the entire dataset each time. It selects only those records that have changed since it last ran. Combined with our Scheduler logic, it creates a system where updates are quickly added to Akita without straining our system or those of our integration partners.

Easy peasy lemon squeazy.

But not really. There are some important edge cases.

The Initial Sync

We have to make sure we catch brand-new Integrations that have not yet synced, so we tweak our SELECT statement to also include any __ Integration Objects that have not yet been synced:


SELECT * FROM IntegrationObjects WHERE SyncedAt <= NOW() - INTERVAL ‘3 HOURS’ OR SyncedAt IS NULL;

This first sync is the costliest—it syncs all data at once. After the initial sync, each update is relatively easy.

Throttling

Akita uses messaging queues extensively (AWS SQS). On occasion, the job queue will be backed up. It is possible that a Retrieve job is queued but doesn’t run before the next evocation of the Scheduler. In this case, there could be duplicate retrieve jobs causing two syncs to run in parallel.

To avoid this, we use throttling to make sure we don’t dispatch the same Retrieve Job more than once in a given time period.

Flexibility

We can adjust the sync frequency of each individual Integration Object (ex. ACME’s Salesforce Tickets). That is we can sync Salesforce Accounts more frequently for customer A than for customer B. This is useful in two situations:


The customer absolutely MUST have their data synced more frequently

Every business is different. Some of our customers have legitimate requirements to have data available in close-to -real time. In these cases, we can adjust their settings to sync a specific Integration Objectmore frequently (by no means in-real-time but fairly frequently).

The customer’s entire dataset changes continuously (making every-3-hour syncs too costly)

Take an analytics tool that provides up-to-the-second statistics for every Account. It may show new results for every Account each time it is queried. Since each record was updated, each sync would require a full re-sync. If there were 100K or 1M Accounts, a full re-sync would mean a never-ending series of API calls.

In this case, we change “every three hours” to “once a day”. This reduces the strain on our servers and our integration partners but still keeps the data fresh enough for almost any Customer Success purpose (building Customer Health Scores, triggering Customer Success Playbooks, or refreshing a Customer Dashboard).

Next week, we’ll look at how our Retriever requests data from our integration partners, handles responses (including errors) and dispatches jobs to the queue for our Processor to analyze.

Related Articles

Best Customer Success Software Best Customer Success Software for Your CS Department in 2024

If you are reading this, you are faced with a real challenge. Either you already manage a customer success function and your existing toolset is not fit for purpose, or you have decided to start a customer success department and want to implement the right software as a solid foundation for your Customer Success Managers […]

Read more
Customer retention metrics Important Customer Retention Metrics and How to Interpret Them

A laser focus on retention is at the heart of any sustainably successful SaaS business. The ability to keep existing customers coming back year after year while at the same time continuing to add new customers has a profound and compounding effect on your revenues. In order to achieve an elite level of retention, you […]

Read more
What is the difference between customer satisfaction and customer retention What is the Difference Between Customer Satisfaction and Customer Retention?

Simply put, while measurable, customer satisfaction is subjective and experiential. Retention, however, is an objective, observable, and measurable phenomenon. You either retain a customer or you don’t. It’s clear. The satisfaction a customer feels, however, can change due to a real issue with the software or how that customer is being supported, but it can […]

Read more
Gross Retention vs Net Retention Gross Retention vs. Net Retention: Definitions, Formulas, Examples

As both gross retention and net retention are crucial metrics in evaluating revenue dynamics in SaaS, it is best to perceive them as complementary rather than opposing metrics. By understanding the fundamentals of your company’s customer retention (GRR) and being able to identify the opportunities for upsells and cross-sells (NRR), you are getting a full […]

Read more
How to Calculate customer retention rate How to Calculate Customer Retention Rate? (+ Benefits)

In SaaS, customer retention stands as a key indicator of sustained success. When you understand how to measure and improve customer retention rates, you can secure your business lasting customer loyalty, sustained growth, and a better market position.

Read more
what is retention rate What is Retention Rate and How to Calculate it?

SaaS thrives on ongoing relationships, where delivering value and ensuring continuously high levels of satisfaction are vital for revenue growth and sustainability. Ultimately, when it comes to long-term success, holding onto your customers can be more important than acquiring them. For that reason, you should know what is the retention rate so you can be on […]

Read more
Customer Retention Analysis Customer Retention Analysis: A Guide for SaaS Businesses

The ability to sustainably grow revenues from your existing customer base is essential to thesurvival of SaaS companies in 2024.  To do this, you first need to retain your customers. This requires a deep understanding of what it is about your product, service, or brand that drives customers to continue investing over time.Customer retention analysis […]

Read more
What is customer retention cost What is Customer Retention Cost and How to Reduce It?

It can take time for a SaaS or subscription company to achieve profitability with a customer. As only part of the revenue is realized up front, much effort is required to ensure sufficient future revenue is earned. You may wonder “What is Customer Retention Cost (CRC)?” — It’s the associated cost of serving the customer […]

Read more