How to set up an event logging with Log Transformations

How to set up an event logging with Log Transformations

Purpose of Log Transformations

Setup a logging of events is crucial, when you need to analyse some events in your application.

Log Transformations is an effective tool to easily setup logging of events.

An example of event could be e.g. adding an item to a shopping cart in your e-shop application.

An example of analytics you could be interested in:

  • count for day
  • count for product and day
  • count for geo-region and day

Scenarios of Log Transformations use

Log Transformations could be set in API data pages.

So they are applicable for standard or javascript web application.

  • Standard web application based on HTML is generated using API data pages.
  • Javascript-based web appliaction using API data pages as backend.
  • Special events (e.g. deletion of a shopping cart) possibly doesn’t correspond to any existing data page. It can be logged by special API data page created only for logging purpose and to be called by javascript from web application.

Currently, it is not possible to apply Log Transfromations on workflow data pages.

1. Logging Setup

First we need to decide, which API page and its data we want to log. We need to choose one from API at model browser.

This paricular one gets the data of all timesheets of selected project (from ORIGAM demo application)

2. Create Log transformation

Log transformation takes a data of Data Structure (ProjectWithTimesheet) and transforms it to the data, that are expected by logger.

In transformation we can specify, which data from the structure we want to log.

Create new transformation in your application:

6258729_310x107

An example of xslt transformation

 <?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="1.0"
    xmlns:AS="http://schema.advantages.cz/AsapFunctions"
    xmlns:date="http://exslt.org/dates-and-times" exclude-result-prefixes="AS date">
    
    <xsl:template match="ROOT">
        <ROOT>
            <!-- Add arbitrary infromation
                 - e.g. Name and Surname of currently logged user -->
            <!-- copy whole content into one field as a xml -->
            <LogContext                
                UserBusinessPartnerInfo="{AS:LookupValue(
                        '08d82ef5-df38-48d6-a276-ee5b7f027bd6',
                        AS:ActiveProfileId())}"
                VarData="{AS:NodeToString(.)}" >LOG_ALL
            </LogContext>
        </ROOT>
    </xsl:template>
</xsl:stylesheet>

Data logger expects xml element LogContent which is supplied. All attributes of LogContent with arbitraty names will be accessible at logger.

In our example we send to LogContent 2 attrributes :

Attribute name Description
UserBusinessPartnerInfo Name and surname of currently logged user (make use of LookupValue)
VarData Contains all data of input XML data structure (ProjectWithTimesheets) serialized as a string (make use of AS:NodeToString() function)

This approach with ‘VarData’ logs all the data without any modification.

This approach has an advantage of storing everything (because sometimes the historic data are no more available). Moreover, this paricular transformation is generic and can be used to any API data page.

The disadvantage could be the volume of logged data.

3. Configure “Analytics” logger

Firstly we have to create a log4net logger and add an appender.

Add appender to Log4Net config section

<log4net>
... 
    <logger name="Analytics">
        <level value="INFO" />
        <appender-ref ref="DetailedAnalyticsAppender" />
        <!-- ... more analytics appenders here -->
    </logger>
...
</log4net>

4A. Setup database appender

Now we have to configure an appender. The example expected, that a database table Log exists with columns VarData, PageId and User.

The logging table needn’t to be in main application database. It can be placed even at another database engine.

Setup a database appender

      <appender name="DetailedAnalyticsAppender" type="log4net.Appender.AdoNetAppender">
          <filter type="log4net.Filter.StringMatchFilter">
              <stringToMatch value="LOG_ALL" />
          </filter>
          <filter type="log4net.Filter.DenyAllFilter" />
          <threshold value="ALL" />
          <bufferSize value="1" />
          <connectionType value="System.Data.SqlClient.SqlConnection, System.Data, Version=1.0.3300.0, Culture=neutral, PublicKeyToken=b77a5c561934e089" />
          <connectionString value="Data Source=<sql server instance name>;Initial Catalog=<db name>;User ID=<username>;Password=<passwd>" />
          <commandText value="INSERT INTO [dbo].[Log] ([VarData],[PageId],[User]) VALUES
            ( @VarData,
              CASE WHEN @PageId = '(null)' THEN NULL ELSE @PageId END),
            , @User)" />
          <parameter>
              <parameterName value="@VarData" />
              <dbType value="string" />
                <size value="-1" />
                <layout type="log4net.Layout.PatternLayout">
                  <conversionPattern value="%property{VarData}" />
              </layout>
          </parameter>
          <parameter>
              <parameterName value="@User" />
              <dbType value="String" />
              <size value="50" />
              <layout type="log4net.Layout.PatternLayout">
                  <conversionPattern value="%property{UserBusinessPartnerInfo}" />
              </layout>
          </parameter>
          <parameter>
              <parameterName value="@PageId" />
              <dbType value="String" />
              <size value="40" />
              <layout type="log4net.Layout.PatternLayout">
                  <conversionPattern value="%property{AsapPageId}" />
              </layout>
          </parameter>
      </appender>

Parameter elements map attributes from LogContext into Log4Net parameters, that can be used in commandText insert statement.

The following element specify for which context store the appender will apply.

<stringToMatch value="LOG_ALL" />

Besides the attributes that we have passed in log transformation, LogContent containts more build-in automatically generated attributes. To log only some of those paameters and nothing more, we wouldn’t need a log transformation at all.

Attribute name Description
Parameter_<parameter name> Each data page url parameter is automatically added to log context and prepended with “Parameter_” prefix.
ContentType Mime type of the API data page content type.
HttpMethod GET/POST/PUT/…
RawUrl Request url.(raw form)
Url Request url
UrlReferrer Value of url referrer sent in request http header.
UserAgent Value of UserAgent sent in request http header.
Browser Browser name parsed from UserAgent.
BrowserVersion Browser version parsed from UserAgent.
UserHostAddress Client IP address.
UserHostName Domain name resolved from IP address.
UserLanuages ???

4B. Setup file appender

File appender configuration

      <appender name="DetailedAnalyticsAppender" type="log4net.Appender.RollingFileAppender">
          <file value="<path-to-log>" />
          <appendToFile value="true" />
          <rollingStyle value="Date" />
          <datePattern value="yyyyMMdd" />
          <filter type="log4net.Filter.StringMatchFilter">
              <stringToMatch value="LOG_ALL" />
          </filter>          
          <filter type="log4net.Filter.DenyAllFilter" />
          <threshold value="ALL" />
          <bufferSize value="1" />
          
          <layout type="log4net.Layout.PatternLayout">
            <conversionPattern value="'%date','%property{AsapPageId}','%property{UserBusinessPartnerInfo}','%property{VarData}'%newline" />
          </layout>
      </appender>