View source for Reducing Data Usage and Optimising your OdataLink Feeds

= Overview =

To keep OdataLink subscriptions affordable and predictable for everyone, OdataLink has introduced a [https://odatalink.com/fair-use-policy '''Fair Use Policy'''] with a '''''Monthly Download Allowance per licensed Data File'''''.

The guidance below applies to ''all Connected Accounting Systems'', namely Xero, Xero Practice Manager, MYOB and QuickBooks, but is most relevant to Xero given the new upstream egress charges. Following these steps will:

* Reduce your monthly download volume,
* Improve refresh performance, and
* Help you stay within your allowance — avoiding optimisation requests or custom pricing.

You can monitor your current usage at any time in the [https://app.odatalink.com OdataLink Admin Portal] under the '''Usage''' tab in Model and Data File details.

<br />

== For everyone (accountants, bookkeepers, business users) ==

The following changes can be applied directly in your OdataLink Admin Portal or in your reporting software:

=== 1. Reduce automatic refresh frequency ===

Most reports do not need to refresh every hour, every day, or on weekends. Each scheduled refresh re-pulls data and counts toward your allowance.

* Match the refresh frequency to how often you actually look at the report.
* Disable refreshes outside business hours and on weekends if the report is not viewed then.
* For monthly reports, a daily or weekly refresh is usually sufficient.

=== 2. Apply a Cut-off Date (Hybrid Archive) on your model ===

The Cut-off Date setting tells OdataLink to treat older data as "archived". Archived data is downloaded once and reused, while only recent data is refreshed on each pull. 

This method dramatically reduces the volume retrieved on every refresh.

[https://odatalink.com/2025-03-increase-performance-of-your-odatalink-odata-feeds-using-the-cut-off-date-settings/ How to use the Cut-off Date setting →]

=== 3. Set a Default Date Range on your model ===

If you never report on data older than, for example, two financial years, you can configure the model to ignore data outside that window entirely. 

This applies a date filter at the source and reduces every download.

[[Default Date Range|How to set a default date range →]]

=== 4. Enable Polling for Changes ===

Polling allows OdataLink to detect and pull only the records that have changed in your accounting file, rather than re-downloading the full dataset on every refresh.

[[Configure OdataLink to poll for changes|How to configure polling →]]

<br />

== For Power BI, Fabric, and Excel developers ==

The following optimisation techniques require working inside Power Query or transformation queries in your reporting software.

=== 5. Push date filters down to the OData feed level (query folding) ===

Filtering data '''after''' it has been downloaded does not reduce egress. The records have already been pulled from the company file. 

Filtering must happen at the '''Navigation step''' in Power Query so that the filter is sent to OdataLink as part of the OData query (`$filter=...`) and only the matching rows are returned.

# Apply date filters as early as possible in the query — directly on the source table, before any merges, joins, or column expansions.
# Avoid applying `Table.SelectRows` over a downloaded table when an equivalent filter can be applied at the navigation step.
# Verify folding by right-clicking a step in Power Query → **View Native Query**. If the option is greyed out, the filter is no longer folding and is being applied client-side.

[[Model Date Filters and Report Date Filters| Model Date Filters vs Report Date Filters →]]

=== 6. Use Incremental Refresh for time-series data ===

For any dataset keyed or based on a date, datetime, or timestamp column, configure '''Incremental Refresh''' so that only new or changed partitions are pulled on each refresh and not the entire history.

* '''Power BI Premium / Fabric:''' native incremental refresh on semantic models.
* '''Dataflow Gen2 (Fabric):''' [https://learn.microsoft.com/en-us/fabric/data-factory/dataflow-gen2-incremental-refresh Dataflow Gen2 incremental refresh →]
* '''Power BI Pro:''' incremental refresh is supported but requires care to ensure the date filter folds back to the OData source. Test with '''View Native Query''' before publishing.

<span style="color:navy"><blockquote>'''Tip:''' OData feeds can be staged into a Fabric Dataflow Gen2, then consumed downstream by Power BI, Excel, Power Apps, and Power Automations. 
This means OdataLink (and therefore Xero) is queried '''once per refresh''', not once per consumer. 
This change can significantly cut total egress when multiple reports share the same data.
</blockquote></span>
<br />

== For SQL, SSIS, and Fabric Developers ==

For users staging OdataLink data into SQL Server, Azure SQL, Fabric Lakehouse / Warehouse, or other persistent stores via SSIS, Azure Data Factory (ADF), or Fabric Data Pipelines.

=== 7. Stage once, pull indefinitely ===

The single biggest egress saving comes from removing duplicate downloads. Avoid having multiple reports, models, or tools all pulling the same OData entity directly. Instead:

# Pull each OData entity into a staging table (SQL Server, Lakehouse, Warehouse) **once per refresh cycle**.
# Point all downstream consumers (Power BI semantic models, Excel reports, SSRS, Fabric notebooks) at the staging layer — not at OdataLink.

This collapses N reports × M refreshes into a single retrieval per cycle.

=== 8. Implement true incremental loads with `$filter` ===

When loading data into a SQL DB or an Azure Lakehouse, do not truncate-and-reload on every run. Use a high-watermark column (e.g. `LastModifiedDateTime`, `UpdatedDateUTC`) and pass it into the OData URL via the `$filter` parameter so OdataLink only returns changed rows.

Example pattern:
<code>GET .../Invoices?$filter=LastModifiedDateTime gt 2026-05-01T00:00:00Z</code>

# Store the maximum `LastModifiedDateTime` from the previous successful load in a control table.
# Use that value as the lower bound for the next pull.
# Merge (UPSERT) the result into the target table on the natural key.

<span style="color:navy"><blockquote>This pattern works in **SSIS** (OData Source component, parameterised URL), **Azure Data Factory** / **Fabric Data Pipelines** (Copy Activity with dynamic OData query), and any custom .NET / Python loader.</blockquote></span>

[[OData Feed URLs| OdataLink OData URL structure and query parameters →]]

=== 9. Select only the columns you need with `$select` ===

OData supports projection via `$select=Col1,Col2,...`. By default, retrieving an entity returns every column, many of which are wide free-text fields that you may never report on. Restricting projection at the URL level reduces row size, network transfer, and data file egress.

Example:
<code>GET .../Invoices?$select=InvoiceID,InvoiceNumber,Date,Total,Status</code>

=== 10. Use `$top` and pagination deliberately ===

For initial historical loads, paginate with `$top` and `$skip` (or follow the `@odata.nextLink`) at a controlled rate. Avoid full-table re-pulls in production schedules — full re-loads should be a one-time backfill, not a recurring job.

=== 11. Schedule pipeline runs during off-peak windows and align cycles ===

* Stagger pipeline schedules so that a single Xero tenant is not being hammered by overlapping pipelines.
* Align pipeline frequency with the underlying business cycle. Daily General Ledger snapshots rarely need sub-hourly retrieval.
* For Fabric Data Pipelines, use **dependency-based triggers** rather than fixed schedules — downstream pipelines run only when the upstream load actually produced new rows.

=== 12. Cache reference data separately ===

Reference entities (Chart of Accounts, Tax Rates, Tracking Categories, Contacts where stable) change far less frequently than transactional entities (Invoices, Bills, Bank Transactions). Refresh reference tables on a slower cadence (daily or weekly) and transactional tables on a faster cadence — instead of refreshing everything together.

<br>

== Quick checklist ==

* [ ] Refresh schedule matches how often the report is actually used
* [ ] Weekend / out-of-hours refreshes disabled where not needed
* [ ] Cut-off Date configured on the model
* [ ] Default Date Range configured on the model
* [ ] Polling enabled
* [ ] Date filters applied at the OData navigation step (not post-download)
* [ ] Query folding verified via **View Native Query**
* [ ] Incremental refresh enabled for time-series datasets
* [ ] Pipelines stage data once into SQL / Lakehouse before downstream consumption
* [ ] `$filter`-based incremental loads with a high-watermark column
* [ ] `$select` used to limit columns retrieved
* [ ] Reference and transactional data refreshed on separate cadences

<br>

== Need help? ==

If you have followed the steps above and your usage is still high, contact OdataLink support. We can review your model configuration, refresh patterns, and pipeline design, and recommend further optimisations before any custom pricing applies.

<br>

== Related Topics ==

* [https://odatalink.com/fair-use-policy OdataLink Fair Use Policy]
* [https://app.odatalink.com OdataLink Admin Portal — Usage tab]
<!-- * [https://developer.xero.com/pricing Xero Developer Pricing]
* [https://developer.xero.com/faq/pricing-and-policy-updates Xero Pricing and Policy Updates FAQ]-->