Reducing Data Usage and Optimising your OdataLink Feeds: Difference between revisions

From OdataLink
Jump to navigation Jump to search
Ash (talk | contribs)
mNo edit summary
Ash (talk | contribs)
mNo edit summary
Line 18: Line 18:
This includes data retrieved by [[Microsoft Power BI|Power BI]], [[Microsoft Excel|Excel]], or [[OData Client Compatibility|other]] data pipeline and reporting tools, either via automatic scheduled refreshes or manual report refresh.
This includes data retrieved by [[Microsoft Power BI|Power BI]], [[Microsoft Excel|Excel]], or [[OData Client Compatibility|other]] data pipeline and reporting tools, either via automatic scheduled refreshes or manual report refresh.


= Pick your Optimisation Layer ==
= Pick your Optimisation Layer =
Optimisation techniques are grouped by the tooling you use. Pick the page that matches your role — or read all three if your organisation spans several layers.
Optimisation techniques are grouped by the tooling you use. Pick the page that matches your role — or read all three if your organisation spans several layers.


Line 28: Line 28:
| [[How to Reduce my OdataLink Usage - For Advanced Users]] || Power BI, Fabric, and Excel report developers (Power Query / M) || Query folding, OData <code>$filter</code> operators, <code>$select</code> projection, incremental refresh
| [[How to Reduce my OdataLink Usage - For Advanced Users]] || Power BI, Fabric, and Excel report developers (Power Query / M) || Query folding, OData <code>$filter</code> operators, <code>$select</code> projection, incremental refresh
|-
|-
| [[How to Reduce my OdataLink Usage - For Power Users]] || Data engineering teams (SSIS, ADF, Fabric Data Pipelines, Fabric Dataflow Gen2) || Stage-once-serve-many, watermarked incremental loads, historical backfill strategy, scheduling discipline, end-to-end cost monitoring
| [[How to Reduce my OdataLink Usage - For Power Users]] || Data engineering teams (SSIS, ADF, Fabric Data Pipelines, Fabric Dataflow Legacy/Gen2) || Stage-once-serve-many, watermarked incremental loads, historical backfill strategy, scheduling discipline, end-to-end cost monitoring
|}
|}


Line 34: Line 34:


<br />
<br />
 
<!--
== For everyone (accountants, bookkeepers, business users) ==
== For everyone (accountants, bookkeepers, business users) ==


Line 57: Line 57:


<br>
<br>
 
-->
== Quick checklist ==
== Quick checklist ==
A cross-cutting checklist drawn from all three pages. Detailed steps and tool-specific guidance are in the linked pages above.
A cross-cutting checklist drawn from all three pages. Detailed steps and tool-specific guidance are in the linked pages above.
Line 69: Line 69:
* [ ] Polling enabled
* [ ] Polling enabled


'''Power BI, Fabric, and Excel developers:'''
'''Advanced Users:'''
* [ ] Date filters applied at the OData navigation step (not post-download)
* [ ] Date filters applied at the OData navigation step (not post-download)
* [ ] Query folding verified via '''View Native Query'''
* [ ] Query folding verified via '''View Native Query'''
Line 75: Line 75:
* [ ] Incremental refresh enabled for time-series datasets
* [ ] Incremental refresh enabled for time-series datasets


'''Data engineering teams:'''
'''Azure/Fabric/Data Pipeline developers:'''
* [ ] OdataLink data staged once into SQL / Lakehouse / Warehouse / Dataflow Gen2; downstream consumers read from the staging layer
* [ ] OdataLink data staged once into SQL / Lakehouse / Warehouse / Dataflow Gen2; downstream consumers read from the staging layer
* [ ] All recurring loads use <code>$filter</code> with a high-watermark column
* [ ] All recurring loads use <code>$filter</code> with a high-watermark column

Revision as of 02:35, 26 May 2026

Overview

To keep OdataLink subscriptions affordable and predictable for everyone, we've introduced a Fair Use Policy. This policy sets a monthly download limit for data retrieved from your connected data file(s).

The guidance below applies to all models connecting to Xero, Xero Practice Manager, MYOB and QuickBooks. Following these steps will:

  • Reduce your monthly download volume; and
  • Improve refresh performance

You can monitor your current usage at any time in your OdataLink Admin Portal → Usage Stats.

See How to track my OdataLink Usage for usage broken down by Data file, Model, and User.


What is Usage

Usage is defined as the volume of data OdataLink retrieves from your connected data files each month.

This includes data retrieved by Power BI, Excel, or other data pipeline and reporting tools, either via automatic scheduled refreshes or manual report refresh.

Pick your Optimisation Layer

Optimisation techniques are grouped by the tooling you use. Pick the page that matches your role — or read all three if your organisation spans several layers.

Page Audience What it covers
How to Reduce my OdataLink Usage - For Everyday Users Accountants, bookkeepers, business users Refresh frequency, model settings (Cut-off Date, Default Date Range, Polling), monitoring
How to Reduce my OdataLink Usage - For Advanced Users Power BI, Fabric, and Excel report developers (Power Query / M) Query folding, OData $filter operators, $select projection, incremental refresh
How to Reduce my OdataLink Usage - For Power Users Data engineering teams (SSIS, ADF, Fabric Data Pipelines, Fabric Dataflow Legacy/Gen2) Stage-once-serve-many, watermarked incremental loads, historical backfill strategy, scheduling discipline, end-to-end cost monitoring

Tip: Start with the Everyday Users page regardless of your role. The model-level settings described there apply to every refresh — automatic or manual, report or pipeline — and deliver the largest reduction with the least effort.


Quick checklist

A cross-cutting checklist drawn from all three pages. Detailed steps and tool-specific guidance are in the linked pages above.

Everyone:

  • [ ] Refresh schedule matches how often the report is actually used
  • [ ] Weekend / out-of-hours refreshes disabled where not needed
  • [ ] Archive Type set to Live (with Archive) on the model
  • [ ] Cut-off Date or rolling window configured
  • [ ] Default Date Range configured to match your reporting window
  • [ ] Polling enabled

Advanced Users:

  • [ ] Date filters applied at the OData navigation step (not post-download)
  • [ ] Query folding verified via View Native Query
  • [ ] $select used to limit columns retrieved
  • [ ] Incremental refresh enabled for time-series datasets

Azure/Fabric/Data Pipeline developers:

  • [ ] OdataLink data staged once into SQL / Lakehouse / Warehouse / Dataflow Gen2; downstream consumers read from the staging layer
  • [ ] All recurring loads use $filter with a high-watermark column
  • [ ] $select used on every entity, including the watermark column
  • [ ] Full-table backfills are one-time, not recurring
  • [ ] Pipeline schedules staggered and aligned with the business cycle
  • [ ] Reference and transactional data refreshed on separate cadences
  • [ ] Both OdataLink egress and Azure / Fabric compute cost monitored


Need help?

If you have followed the steps above and your usage is still high, contact OdataLink support. We can review your model configuration, refresh patterns, and pipeline design, and recommend further optimisations to bring your account usage within limits.


Related Topics