Security teams in government and financial services face a persistent tension between investigative completeness and operating cost. Regulatory expectations, incident response obligations, and modern threat hunting all demand that organisations retain a broad sweep of telemetry for long periods, yet high-volume logs such as firewall traffic, proxy records, and verbose application diagnostics can dominate a monitoring budget within weeks. The result is a difficult compromise, where teams either drop valuable signal to control spend or pay premium ingestion rates for data they rarely query interactively.
Azure Monitor and Log Analytics address this problem through a tiered approach to data, allowing you to match each stream to a plan that reflects how you actually use it. Rather than treating all telemetry as equally valuable and equally expensive, the platform lets you separate the data you investigate daily from the long-tail records you keep for compliance and occasional forensic reach. Understanding these tiers is the foundation of a security data lake that remains both affordable and defensible.
Log Analytics organises data using table plans, and each table in a workspace can be assigned to one of three options: Analytics, Basic, or Auxiliary. The Analytics plan is the familiar full-fidelity tier, supporting interactive queries, scheduled alert rules, dashboards, and the complete Kusto Query Language experience. It suits the signals your analysts touch constantly, such as sign-in logs, endpoint detections, and the correlated events that drive your alerting pipeline.
The Basic and Auxiliary plans exist for data with a different usage profile, typically high in volume but low in the frequency of interactive access. Basic logs reduce ingestion cost in exchange for a constrained query surface and limited alerting, which fits verbose logs that you occasionally search during an investigation. Auxiliary logs push this further still, offering the lowest ingestion cost for very high-volume, lower-fidelity telemetry that you want available for search jobs and summarisation, but which does not justify the premium of the Analytics tier.
The practical work of rightsizing begins with classifying your sources by how they contribute to detection and investigation. Identity events, privileged access activity, and endpoint alerts generally belong in the Analytics plan because they feed real-time correlation and demand rich querying. Network flow logs, DNS query records, and chatty platform diagnostics are often better candidates for Basic or Auxiliary, since their value is realised during scoping and forensic reconstruction rather than minute-by-minute monitoring.
This classification should be deliberate and documented, because a misplaced table can either inflate cost or quietly remove a capability your analysts assumed was present. A useful discipline is to review each table against three questions: do you alert on it, do you query it interactively each week, and would you need it during a breach that is discovered months later. Sources that answer no to the first two but yes to the third are strong candidates for a lower-cost tier paired with extended retention, preserving investigative reach without carrying premium ingestion charges across the whole estate.
Retention in Log Analytics separates interactive retention, during which data is immediately queryable, from long-term retention, where data is held more cheaply and accessed through specific mechanisms. Analytics tables provide a default interactive retention period of 30 days, and workspaces with Microsoft Sentinel enabled receive 90 days of retention at no additional charge. Beyond the interactive window, total retention can be extended up to 12 years, which allows you to satisfy lengthy compliance mandates without keeping every record hot and expensive.
Accessing data held in long-term retention relies on search jobs and data restore rather than live queries, and this difference shapes how you plan investigations. A search job runs asynchronously across archived data and returns matching records to a new table, which is well suited to the kind of retrospective hunt that follows a late-discovered compromise. By combining lower-cost ingestion tiers with extended long-term retention, you keep the long tail of security telemetry available for months or years while paying interactive rates only for the data that genuinely warrants immediate access.
Organisations bound by the Australian Government Information Security Manual and sector-specific obligations should treat tiering decisions as part of their broader logging and monitoring posture rather than a purely financial exercise. Event logging guidance consistently emphasises retaining sufficient records to support detection and investigation, so any move to a lower-cost tier must preserve the fields, sources, and retention windows that your detection strategy and regulatory commitments depend upon. Mapping each table to the relevant control objective helps ensure that cost optimisation never silently degrades your ability to investigate or report an incident.
It is also worth validating assumptions through testing before committing a production estate to a particular configuration. Confirm that alert rules you rely upon are compatible with the chosen plan, rehearse a search job against archived data so your responders understand the latency involved, and periodically revisit classifications as data volumes and threat patterns change. With that governance in place, the tiered model becomes a sustainable way to run a comprehensive security data lake, keeping high-value signal responsive while holding the long tail of telemetry affordably and within reach.

Level 7, 12 St Georges Tce
Perth WA 6000
[email protected]
Ph 1300 NOVATA

In the spirit of reconciliation Novata Solutions acknowledges the Traditional Custodians of country throughout Australia and their connections to land, sea and community. We pay our respect to their Elders past and present and extend that respect to all Aboriginal and Torres Strait Islander peoples today. This land always was, and always will be Aboriginal Land.

Novata Solutions is committed to embracing diversity and eliminating all forms of discrimination through education. We welcomes all people and is respectful of individual identities.