How to set up an event logging with Log Transformations

Setting up a logging of events is crucial when you need to analyse some events in your application.

Log Transformations is an effective tool to easily set up the logging of events.

An example of an event could be e.g. adding an item to a shopping cart in your e-shop application.

An example of analytics you could be interested in:

  • count for the day
  • count for product and day
  • count for geo-region and day

Scenarios of Log Transformations use

Log Transformations could be set in API data pages.

So they are applicable for standard or javascript web applications.

  • Standard web application based on HTML is generated using API data pages.
  • Javascript-based web application using API data pages as the backend.
  • Special events (e.g. deletion of a shopping cart) possibly don’t correspond to any existing data page. It can be logged by a special API data page created only for logging purposes and to be called by javascript from a web application.

Currently, it is not possible to apply Log transformations on workflow data pages.

1. Logging Setup

First, we need to decide, which API page and its data we want to log. We need to choose one from the API at the model browser.

This particular one gets the data of all timesheets of selected projects (from ORIGAM demo application)

2. Create Log transformation

Log transformation takes data of Data Structure (ProjectWithTimesheet) and transforms it to the data, that are expected by the logger.

In transformation, we can specify, which data from the structure we want to log.

Create a new transformation in your application:

6258729_310x107

An example of XSLT transformation

 <?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="1.0"
    xmlns:AS="http://schema.advantages.cz/AsapFunctions"
    xmlns:date="http://exslt.org/dates-and-times" exclude-result-prefixes="AS date">
    
    <xsl:template match="ROOT">
        <ROOT>
            <!-- Add arbitrary information
                 - e.g. Name and Surname of currently logged user -->
            <!-- copy whole content into one field as a xml -->
            <LogContext                
                UserBusinessPartnerInfo="{AS:LookupValue(
                        '08d82ef5-df38-48d6-a276-ee5b7f027bd6',
                        AS:ActiveProfileId())}"
                VarData="{AS:NodeToString(.)}" >LOG_ALL
            </LogContext>
        </ROOT>
    </xsl:template>
</xsl:stylesheet>

The data logger expects XML element LogContent which is supplied. All attributes of LogContent with arbitrary names will be accessible at logger.

In our example, we send to LogContent 2 attributes :

Attribute name Description
UserBusinessPartnerInfo Name and surname of currently logged user (make use of LookupValue)
VarData Contains all data of input XML data structure (ProjectWithTimesheets) serialized as a string (make use of AS:NodeToString() function)

This approach with ‘VarData’ logs all the data without any modification.

This approach has the advantage of storing everything (because sometimes the historical data are no more available). Moreover, this particular transformation is generic and can be used on any API data page.

The disadvantage could be the volume of logged data.

3. Configure “Analytics” logger

Firstly we have to create a log4net logger and add an appender.

Add appender to Log4Net config section

<log4net>
... 
    <logger name="Analytics">
        <level value="INFO" />
        <appender-ref ref="DetailedAnalyticsAppender" />
        <!-- ... more analytics appenders here -->
    </logger>
...
</log4net>

4A. Setup database appender

Now we have to configure an appender. The example expected, is that a database table Log exists with columns VarData, PageId and User.

The logging table needn’t be in the main application database. It can be placed even in another database engine.

Setup a database appender

      <appender name="DetailedAnalyticsAppender" type="log4net.Appender.AdoNetAppender">
          <filter type="log4net.Filter.StringMatchFilter">
              <stringToMatch value="LOG_ALL" />
          </filter>
          <filter type="log4net.Filter.DenyAllFilter" />
          <threshold value="ALL" />
          <bufferSize value="1" />
          <connectionType value="System.Data.SqlClient.SqlConnection, System.Data, Version=1.0.3300.0, Culture=neutral, PublicKeyToken=b77a5c561934e089" />
          <connectionString value="Data Source=<sql server instance name>;Initial Catalog=<db name>;User ID=<username>;Password=<passwd>" />
          <commandText value="INSERT INTO [dbo].[Log] ([VarData],[PageId],[User]) VALUES
            ( @VarData,
              CASE WHEN @PageId = '(null)' THEN NULL ELSE @PageId END),
            , @User)" />
          <parameter>
              <parameterName value="@VarData" />
              <dbType value="string" />
                <size value="-1" />
                <layout type="log4net.Layout.PatternLayout">
                  <conversionPattern value="%property{VarData}" />
              </layout>
          </parameter>
          <parameter>
              <parameterName value="@User" />
              <dbType value="String" />
              <size value="50" />
              <layout type="log4net.Layout.PatternLayout">
                  <conversionPattern value="%property{UserBusinessPartnerInfo}" />
              </layout>
          </parameter>
          <parameter>
              <parameterName value="@PageId" />
              <dbType value="String" />
              <size value="40" />
              <layout type="log4net.Layout.PatternLayout">
                  <conversionPattern value="%property{AsapPageId}" />
              </layout>
          </parameter>
      </appender>

Parameter elements map attributes from LogContext into Log4Net parameters, that can be used in commandText insert statement.

The following element specifies for which context stores the appender will apply.

<stringToMatch value="LOG_ALL" />

Besides the attributes that we have passed in log transformation, LogContent contents more built-in automatically generated attributes. To log only some of those parameters and nothing more, we wouldn’t need a log transformation at all.

Attribute name Description
Parameter_<parameter name> Each data page url parameter is automatically added to log context and prepended with “Parameter_” prefix.
ContentType Mime type of the API data page content type.
HttpMethod GET/POST/PUT/…
RawUrl Request url.(raw form)
Url Request url
UrlReferrer Value of url referrer sent in request http header.
UserAgent Value of UserAgent sent in request HTTP header.
Browser Browser name parsed from UserAgent.
BrowserVersion Browser version parsed from UserAgent.
UserHostAddress Client IP address.
UserHostName Domain name resolved from IP address.
UserLanuages ???

4B. Setup file appender

File appender configuration

      <appender name="DetailedAnalyticsAppender" type="log4net.Appender.RollingFileAppender">
          <file value="<path-to-log>" />
          <appendToFile value="true" />
          <rollingStyle value="Date" />
          <datePattern value="yyyyMMdd" />
          <filter type="log4net.Filter.StringMatchFilter">
              <stringToMatch value="LOG_ALL" />
          </filter>          
          <filter type="log4net.Filter.DenyAllFilter" />
          <threshold value="ALL" />
          <bufferSize value="1" />
          
          <layout type="log4net.Layout.PatternLayout">
            <conversionPattern value="'%date','%property{AsapPageId}','%property{UserBusinessPartnerInfo}','%property{VarData}'%newline" />
          </layout>
      </appender>