How Frequently Akita App Syncs Your Data?

How Frequently Akita App Syncs Your Data

In a previous post, we looked at How We Built More Than 100 Customer Success-Integrations between Akita and apps like Hubspot, Salesforce, Intercom and Zendesk. We outlined our 3-step process for keeping data in sync: Schedule, Retrieve, and Process.

The Scheduler kicks everything off and is responsible for determining which data needs retrieving at any given time.

Let’s look at a customer (ACME) who has connected their Salesforce account. We can sync many types of data (or Interaction Types) from Salesforce including:

  • Accounts
  • Contacts
  • Opportunities
  • Tasks
  • Notes
  • Users

By default, we sync each type of Salesforce data every 180 minutes. This may be overkill for “Users” (ie. new Salesforce licenses you might rarely add) but is useful for having timely access to new Accounts and Contacts in Akita.

We schedule the retrieval of each different Interaction Type for ACME (an Integration Object) separately. This simplified SQL statement allows us to select the Integration Objects that need syncing:

SELECT * FROM IntegrationObjects WHERE SyncedAt <= NOW() - INTERVAL ‘3 HOURS’;

The Scheduler runs every couple of minutes, finding any Integration Objects (ex. ACME’s > Salesforce > Accounts) that need syncing, creating a Retrieve Job and dispatching it to a queue. When the Retriever successfully executes the Retrieve Job, it will update the SyncedAt property to the current time.

The Retriever doesn’t re-sync the entire dataset each time. It selects only those records that have changed since it last ran. Combined with our Scheduler logic, it creates a system where updates are quickly added to Akita without straining our system or those of our integration partners.

Easy peasy lemon squeezy.

But not really. There are some important edge cases.

The Initial Sync

We have to make sure we catch brand-new Integrations that have not yet synced, so we tweak our SELECT statement to also include any __ Integration Objects that have not yet been synced:


SELECT * FROM IntegrationObjects WHERE SyncedAt <= NOW() - INTERVAL ‘3 HOURS’ OR SyncedAt IS NULL;

This first sync is the costliest—it syncs all data at once. After the initial sync, each update is relatively easy.

Throttling

Akita uses messaging queues extensively (AWS SQS). On occasion, the job queue will be backed up. It is possible that a Retrieve job is queued but doesn’t run before the next evocation of the Scheduler. In this case, there could be duplicate retrieve jobs causing two syncs to run in parallel.

To avoid this, we use throttling to make sure we don’t dispatch the same Retrieve Job more than once in a given time period.

Flexibility

We can adjust the sync frequency of each individual Integration Object (ex. ACME’s Salesforce Tickets). That is we can sync Salesforce Accounts more frequently for customer A than for customer B. This is useful in two situations:

The customer absolutely MUST have their data synced more frequently

Every business is different. Some of our customers have legitimate requirements to have data available in close-to -real time. In these cases, we can adjust their settings to sync a specific Integration Objectmore frequently (by no means in-real-time but fairly frequently).

The customer’s entire dataset changes continuously (making every-3-hour syncs too costly)

Take an analytics tool that provides up-to-the-second statistics for every Account. It may show new results for every Account each time it is queried. Since each record was updated, each sync would require a full re-sync. If there were 100K or 1M Accounts, a full re-sync would mean a never-ending series of API calls.

In this case, we change “every three hours” to “once a day”. This reduces the strain on our servers and our integration partners but still keeps the data fresh enough for almost any Customer Success purpose (building Customer Health Scores, triggering Customer Success Playbooks, or refreshing a Customer Dashboard).

Next week, we’ll look at how our Retriever requests data from our integration partners, handles responses (including errors) and dispatches jobs to the queue for our Processor to analyze.

Related Articles

Best customer segmentation software Best Customer Segmentation Software to Utilize in 2024

Understanding customers at scale has come a long way. Back in the day, companies used simple attributes to carve up their customer base in some way to apply different resources to support and/or expand those customers. Boy was it blunt though! Generally, it never got far beyond segmenting your account list by their spending or […]

Read more
best customer success software Best Customer Success Software for Your CS Department in 2024

If you are reading this, you are faced with a real challenge. Either you already manage a customer success function and your existing toolset is not fit for purpose, or you have decided to start a customer success department and want to implement the right software as a solid foundation for your Customer Success Managers […]

Read more
top customer retention software Top Customer Retention Software to Reduce Churn in 2024

Figuring out the right customer retention management software to help manage your Customer Retention goals can be tricky. It really comes down to what your specific goals are, what type of market you serve, the customer data you retain and where that data resides. There are some specific types of tools you should consider. Customer […]

Read more
Expansion revenue Expansion Revenue: Definitions, Metrics, Formulas and Tips

Recurring revenue is essential for any SaaS business. You are probably aware of the fact that customer acquisition is often very expensive, with the risk of getting only a few lukewarm leads. Now, is it possible to cut customer acquisition costs and still generate more recurring revenue? This is precisely what expansion revenue brings to […]

Read more
customer churn rate Customer Churn Rate: Definition, Formulas and Benefits

Although customer churn, losing some customers over time, is perceived as inevitable in certain cases, it is also undesirable among companies. However, every business experiences churn, regardless of its size, quality of customer service and the product itself.  What’s important is how your company responds to churn, whether you fully understand why it happens, and […]

Read more
Best customer success blogs 12 Best Customer Success Blogs in 2024 You Should Follow 

With their aims of transforming new customers into loyal ones, nurturing customer relationships during their entire journey, and driving revenue growth through upsells and cross-sells, customer success teams have become one of the most valuable assets of any SaaS company. And, it’s important to be updated on the latest trends, best practices and strategies in […]

Read more
customer churn analysis What is Customer Churn Analysis and How to Perform It? 

Churn is, unfortunately, an inevitable part of any business, no matter its size, reputation, or customer retention efforts. Some businesses are faced with higher and some with lower churn rates, yet what matters most is determining the primary cause of churn to be able to take effective measures to prevent it. This is precisely where […]

Read more
reasons for customer churn Reasons for Customer Churn and Solutions for Some Scenarios

In SaaS, nurturing strong customer relationships is one of the customer retention strategies for business survival and growth. Still, it seems that no matter how hard you work, there will be some customers who simply decide to leave. Although customer churn is inevitable, certain reasons for it can help you predict it and eventually, reduce […]

Read more
customer success goals Different Customer Success Goals: Which One to Choose?

Nurturing customer relationships, assisting and educating customers during onboarding, pushing upsells and expansion…the list goes on and on. Still, these are just some of the responsibilities of the customer success team. 

Read more