Cosmos Feed Processor

The app is used for monitoring purposes. It connects to a CosmosDB collection's change feed (so all new and updated documents) and saves them to ApplicationInsights. From there it can be used in Grafana or Azure Monitor. Currently data about Actions is saved there, as it might be very useful insight apart from apps metrics

The app will read all the new documents coming in from the Cosmos and save them in the format:

{
    name: "streamResult",
    properties: {
        url: <stream url>,
        statusCode: <response status code, if empty then 0>,
        isError: <boolean - if status code bigger or equal to 400>,
        timestamp: <timestamp of moment being saved in CosmosDB>,
        organizationId: <organization ID from PostgreSQL>
    }
}

{
    name: "smsResults",
    properties: {
        recipient: <phone number>,
        timestamp: <timestamp of moment being saved in CosmosDB>,
        organizationId: <organization ID from PostgreSQL>
    }
}

{
    name: "emailResults",
    properties: {
        recipient: <email address>,
        timestamp: <timestamp of moment being saved in CosmosDB>,
        organizationId: <organization ID from PostgreSQL>
    }
}

Each document will be saved as a separate row in ApplicationInsights KQL. Saving happens either every 10 seconds or when buffer size reaches 100.

How does monitoring works in infrastructure

In Azure Monitor it's possible to create an Alert Rule based on either Azure Metric, such as CPU usage or KQL query from Application Insights. Alert Rule have define time period how often they check criteria and how much in the past they look. Once criteria is reached, the Action Group notification is being executed. Currently it's sending email to iotde_ops@proximus.com and gerdy.seynaeve@proximus.com.

Sometimes the library of Application Insights is buffering certain messages and despite forcing emptying it, it might save some logs twice. Therefore we should use distinct correlationId in the query in order to make sure that we won't bill for some action twice.

Examples - count SMS:

customEvents
| where name == 'smsResults'
| project 
    recipient = customDimensions.recipient,
    organizationId = customDimensions.organizationId,
    timestamp = timestamp,
    correlationId = tostring(customDimensions.correlationId)
| distinct correlationId 
| count;

Examples - count SMS per month per orgId

customEvents
 where name == 'smsResults'
 project 
    recipient = customDimensions.recipient,
    organizationId = tostring(customDimensions.organizationId),
    timestamp = timestamp,
    correlationId = tostring(customDimensions.correlationId),
    name = name
 summarize count() by bin(timestamp, 1ms), organizationId
 summarize count() by startofmonth(timestamp), organizationId