Warning: preg_replace(): Compilation failed: invalid range in character class at offset 4 in /home/.sites/946/site2164767/web/blog/wp-content/plugins/crayon-syntax-highlighter/crayon_langs.class.php on line 340 Warning: preg_replace(): Compilation failed: invalid range in character class at offset 4 in /home/.sites/946/site2164767/web/blog/wp-content/plugins/crayon-syntax-highlighter/crayon_langs.class.php on line 340 Warning: preg_replace(): Compilation failed: invalid range in character class at offset 4 in /home/.sites/946/site2164767/web/blog/wp-content/plugins/crayon-syntax-highlighter/crayon_langs.class.php on line 340 Warning: preg_replace(): Compilation failed: invalid range in character class at offset 4 in /home/.sites/946/site2164767/web/blog/wp-content/plugins/crayon-syntax-highlighter/crayon_langs.class.php on line 340 Warning: preg_replace(): Compilation failed: invalid range in character class at offset 4 in /home/.sites/946/site2164767/web/blog/wp-content/plugins/crayon-syntax-highlighter/crayon_langs.class.php on line 340 Warning: preg_replace(): Compilation failed: invalid range in character class at offset 4 in /home/.sites/946/site2164767/web/blog/wp-content/plugins/crayon-syntax-highlighter/crayon_langs.class.php on line 340 Warning: preg_replace(): Compilation failed: invalid range in character class at offset 4 in /home/.sites/946/site2164767/web/blog/wp-content/plugins/crayon-syntax-highlighter/crayon_langs.class.php on line 340 Warning: preg_replace(): Compilation failed: invalid range in character class at offset 4 in /home/.sites/946/site2164767/web/blog/wp-content/plugins/crayon-syntax-highlighter/crayon_langs.class.php on line 340 Warning: preg_replace(): Compilation failed: invalid range in character class at offset 4 in /home/.sites/946/site2164767/web/blog/wp-content/plugins/crayon-syntax-highlighter/crayon_langs.class.php on line 340 Warning: preg_replace(): Compilation failed: invalid range in character class at offset 4 in /home/.sites/946/site2164767/web/blog/wp-content/plugins/crayon-syntax-highlighter/crayon_langs.class.php on line 340 Warning: preg_replace(): Compilation failed: invalid range in character class at offset 4 in /home/.sites/946/site2164767/web/blog/wp-content/plugins/crayon-syntax-highlighter/crayon_langs.class.php on line 340 Warning: preg_replace(): Compilation failed: invalid range in character class at offset 4 in /home/.sites/946/site2164767/web/blog/wp-content/plugins/crayon-syntax-highlighter/crayon_langs.class.php on line 340 Warning: preg_replace(): Compilation failed: invalid range in character class at offset 4 in /home/.sites/946/site2164767/web/blog/wp-content/plugins/crayon-syntax-highlighter/crayon_langs.class.php on line 340 Warning: preg_replace(): Compilation failed: invalid range in character class at offset 4 in /home/.sites/946/site2164767/web/blog/wp-content/plugins/crayon-syntax-highlighter/crayon_langs.class.php on line 340 Warning: preg_replace(): Compilation failed: invalid range in character class at offset 4 in /home/.sites/946/site2164767/web/blog/wp-content/plugins/crayon-syntax-highlighter/crayon_langs.class.php on line 340 Warning: preg_replace(): Compilation failed: invalid range in character class at offset 4 in /home/.sites/946/site2164767/web/blog/wp-content/plugins/crayon-syntax-highlighter/crayon_langs.class.php on line 340 Warning: preg_replace(): Compilation failed: invalid range in character class at offset 4 in /home/.sites/946/site2164767/web/blog/wp-content/plugins/crayon-syntax-highlighter/crayon_langs.class.php on line 340 Warning: preg_replace(): Compilation failed: invalid range in character class at offset 4 in /home/.sites/946/site2164767/web/blog/wp-content/plugins/crayon-syntax-highlighter/crayon_langs.class.php on line 340 Warning: preg_replace(): Compilation failed: invalid range in character class at offset 4 in /home/.sites/946/site2164767/web/blog/wp-content/plugins/crayon-syntax-highlighter/crayon_langs.class.php on line 340 Warning: preg_replace(): Compilation failed: invalid range in character class at offset 4 in /home/.sites/946/site2164767/web/blog/wp-content/plugins/crayon-syntax-highlighter/crayon_langs.class.php on line 340 Warning: preg_replace(): Compilation failed: invalid range in character class at offset 4 in /home/.sites/946/site2164767/web/blog/wp-content/plugins/crayon-syntax-highlighter/crayon_langs.class.php on line 340 Warning: preg_replace(): Compilation failed: invalid range in character class at offset 4 in /home/.sites/946/site2164767/web/blog/wp-content/plugins/crayon-syntax-highlighter/crayon_langs.class.php on line 340 Warning: preg_replace(): Compilation failed: invalid range in character class at offset 4 in /home/.sites/946/site2164767/web/blog/wp-content/plugins/crayon-syntax-highlighter/crayon_langs.class.php on line 340 Warning: preg_replace(): Compilation failed: invalid range in character class at offset 4 in /home/.sites/946/site2164767/web/blog/wp-content/plugins/crayon-syntax-highlighter/crayon_langs.class.php on line 340 Warning: preg_replace(): Compilation failed: invalid range in character class at offset 4 in /home/.sites/946/site2164767/web/blog/wp-content/plugins/crayon-syntax-highlighter/crayon_langs.class.php on line 340 Warning: preg_replace(): Compilation failed: invalid range in character class at offset 4 in /home/.sites/946/site2164767/web/blog/wp-content/plugins/crayon-syntax-highlighter/crayon_langs.class.php on line 340 Warning: preg_replace(): Compilation failed: invalid range in character class at offset 4 in /home/.sites/946/site2164767/web/blog/wp-content/plugins/crayon-syntax-highlighter/crayon_langs.class.php on line 340 Warning: preg_replace(): Compilation failed: invalid range in character class at offset 4 in /home/.sites/946/site2164767/web/blog/wp-content/plugins/crayon-syntax-highlighter/crayon_langs.class.php on line 340 Warning: preg_replace(): Compilation failed: invalid range in character class at offset 4 in /home/.sites/946/site2164767/web/blog/wp-content/plugins/crayon-syntax-highlighter/crayon_langs.class.php on line 340 Warning: preg_replace(): Compilation failed: invalid range in character class at offset 4 in /home/.sites/946/site2164767/web/blog/wp-content/plugins/crayon-syntax-highlighter/crayon_langs.class.php on line 340 Warning: preg_replace(): Compilation failed: invalid range in character class at offset 4 in /home/.sites/946/site2164767/web/blog/wp-content/plugins/crayon-syntax-highlighter/crayon_langs.class.php on line 340 Warning: preg_replace(): Compilation failed: invalid range in character class at offset 4 in /home/.sites/946/site2164767/web/blog/wp-content/plugins/crayon-syntax-highlighter/crayon_langs.class.php on line 340 Warning: preg_replace(): Compilation failed: invalid range in character class at offset 4 in /home/.sites/946/site2164767/web/blog/wp-content/plugins/crayon-syntax-highlighter/crayon_langs.class.php on line 340 Warning: preg_replace(): Compilation failed: invalid range in character class at offset 4 in /home/.sites/946/site2164767/web/blog/wp-content/plugins/crayon-syntax-highlighter/crayon_langs.class.php on line 340 Warning: preg_replace(): Compilation failed: invalid range in character class at offset 4 in /home/.sites/946/site2164767/web/blog/wp-content/plugins/crayon-syntax-highlighter/crayon_langs.class.php on line 340 Warning: preg_replace(): Compilation failed: invalid range in character class at offset 4 in /home/.sites/946/site2164767/web/blog/wp-content/plugins/crayon-syntax-highlighter/crayon_langs.class.php on line 340 Warning: preg_replace(): Compilation failed: invalid range in character class at offset 4 in /home/.sites/946/site2164767/web/blog/wp-content/plugins/crayon-syntax-highlighter/crayon_langs.class.php on line 340 Warning: preg_replace(): Compilation failed: invalid range in character class at offset 4 in /home/.sites/946/site2164767/web/blog/wp-content/plugins/crayon-syntax-highlighter/crayon_langs.class.php on line 340 Warning: preg_replace(): Compilation failed: invalid range in character class at offset 4 in /home/.sites/946/site2164767/web/blog/wp-content/plugins/crayon-syntax-highlighter/crayon_langs.class.php on line 340 Warning: preg_replace(): Compilation failed: invalid range in character class at offset 4 in /home/.sites/946/site2164767/web/blog/wp-content/plugins/crayon-syntax-highlighter/crayon_langs.class.php on line 340 Warning: preg_replace(): Compilation failed: invalid range in character class at offset 4 in /home/.sites/946/site2164767/web/blog/wp-content/plugins/crayon-syntax-highlighter/crayon_langs.class.php on line 340 Warning: preg_replace(): Compilation failed: invalid range in character class at offset 4 in /home/.sites/946/site2164767/web/blog/wp-content/plugins/crayon-syntax-highlighter/crayon_langs.class.php on line 340 Warning: preg_replace(): Compilation failed: invalid range in character class at offset 4 in /home/.sites/946/site2164767/web/blog/wp-content/plugins/crayon-syntax-highlighter/crayon_langs.class.php on line 340 Warning: preg_replace(): Compilation failed: invalid range in character class at offset 4 in /home/.sites/946/site2164767/web/blog/wp-content/plugins/crayon-syntax-highlighter/crayon_langs.class.php on line 340 Warning: preg_replace(): Compilation failed: invalid range in character class at offset 4 in /home/.sites/946/site2164767/web/blog/wp-content/plugins/crayon-syntax-highlighter/crayon_langs.class.php on line 340 Warning: preg_replace(): Compilation failed: invalid range in character class at offset 4 in /home/.sites/946/site2164767/web/blog/wp-content/plugins/crayon-syntax-highlighter/crayon_langs.class.php on line 340 Warning: preg_replace(): Compilation failed: invalid range in character class at offset 4 in /home/.sites/946/site2164767/web/blog/wp-content/plugins/crayon-syntax-highlighter/crayon_langs.class.php on line 340 Warning: preg_replace(): Compilation failed: invalid range in character class at offset 4 in /home/.sites/946/site2164767/web/blog/wp-content/plugins/crayon-syntax-highlighter/crayon_langs.class.php on line 340 Warning: preg_replace(): Compilation failed: invalid range in character class at offset 4 in /home/.sites/946/site2164767/web/blog/wp-content/plugins/crayon-syntax-highlighter/crayon_langs.class.php on line 340 Warning: preg_replace(): Compilation failed: invalid range in character class at offset 4 in /home/.sites/946/site2164767/web/blog/wp-content/plugins/crayon-syntax-highlighter/crayon_langs.class.php on line 340 Warning: preg_replace(): Compilation failed: invalid range in character class at offset 4 in /home/.sites/946/site2164767/web/blog/wp-content/plugins/crayon-syntax-highlighter/crayon_langs.class.php on line 340 Warning: preg_replace(): Compilation failed: invalid range in character class at offset 4 in /home/.sites/946/site2164767/web/blog/wp-content/plugins/crayon-syntax-highlighter/crayon_langs.class.php on line 340 Warning: preg_replace(): Compilation failed: invalid range in character class at offset 4 in /home/.sites/946/site2164767/web/blog/wp-content/plugins/crayon-syntax-highlighter/crayon_langs.class.php on line 340 Warning: preg_replace(): Compilation failed: invalid range in character class at offset 4 in /home/.sites/946/site2164767/web/blog/wp-content/plugins/crayon-syntax-highlighter/crayon_langs.class.php on line 340 Warning: preg_replace(): Compilation failed: invalid range in character class at offset 4 in /home/.sites/946/site2164767/web/blog/wp-content/plugins/crayon-syntax-highlighter/crayon_langs.class.php on line 340 Warning: preg_replace(): Compilation failed: invalid range in character class at offset 4 in /home/.sites/946/site2164767/web/blog/wp-content/plugins/crayon-syntax-highlighter/crayon_langs.class.php on line 340 Warning: preg_replace(): Compilation failed: invalid range in character class at offset 4 in /home/.sites/946/site2164767/web/blog/wp-content/plugins/crayon-syntax-highlighter/crayon_langs.class.php on line 340 Warning: preg_replace(): Compilation failed: invalid range in character class at offset 4 in /home/.sites/946/site2164767/web/blog/wp-content/plugins/crayon-syntax-highlighter/crayon_langs.class.php on line 340 Warning: preg_replace(): Compilation failed: invalid range in character class at offset 4 in /home/.sites/946/site2164767/web/blog/wp-content/plugins/crayon-syntax-highlighter/crayon_langs.class.php on line 340 Warning: preg_replace(): Compilation failed: invalid range in character class at offset 4 in /home/.sites/946/site2164767/web/blog/wp-content/plugins/crayon-syntax-highlighter/crayon_langs.class.php on line 340 Warning: preg_replace(): Compilation failed: invalid range in character class at offset 4 in /home/.sites/946/site2164767/web/blog/wp-content/plugins/crayon-syntax-highlighter/crayon_langs.class.php on line 340 Warning: preg_replace(): Compilation failed: invalid range in character class at offset 4 in /home/.sites/946/site2164767/web/blog/wp-content/plugins/crayon-syntax-highlighter/crayon_langs.class.php on line 340 Warning: preg_replace(): Compilation failed: invalid range in character class at offset 4 in /home/.sites/946/site2164767/web/blog/wp-content/plugins/crayon-syntax-highlighter/crayon_langs.class.php on line 340 Warning: preg_replace(): Compilation failed: invalid range in character class at offset 4 in /home/.sites/946/site2164767/web/blog/wp-content/plugins/crayon-syntax-highlighter/crayon_langs.class.php on line 340 Warning: preg_replace(): Compilation failed: invalid range in character class at offset 4 in /home/.sites/946/site2164767/web/blog/wp-content/plugins/crayon-syntax-highlighter/crayon_langs.class.php on line 340 Warning: preg_replace(): Compilation failed: invalid range in character class at offset 4 in /home/.sites/946/site2164767/web/blog/wp-content/plugins/crayon-syntax-highlighter/crayon_langs.class.php on line 340 Warning: Cannot modify header information - headers already sent by (output started at /home/.sites/946/site2164767/web/blog/wp-content/plugins/crayon-syntax-highlighter/crayon_langs.class.php:340) in /home/.sites/946/site2164767/web/blog/wp-includes/feed-rss2.php on line 8 json – ZoolWay https://blog.zoolway.com Coder, Gamer, Gentleman Wed, 09 Mar 2016 20:05:07 +0000 en-US hourly 1 https://wordpress.org/?v=6.0.9 Logging – Human Readable vs. Machine Parsable https://blog.zoolway.com/logging-human-readable-vs-machine-parsable https://blog.zoolway.com/logging-human-readable-vs-machine-parsable#comments Wed, 09 Mar 2016 19:53:46 +0000 http://blog.zoolway.com/?p=103 I have often encouraged other to implement proper logging even in the smallest projects. My .NET MVVM template available at GitHub comes with log4net per default. There is no need to reinvent the wheel. Besides log4net there is NLog and other frameworks. Most can target many diverse storages like files (with automatic rolling), the Windows event log, UDP ports and more. This way we get some decent log files.

Human Readable

With some log file rolling and different log levels it is mostly a good idea to log data which can be important, especially in production debugging. When you get the users to send you the log file or a feedback tool does that for them, great. Now you got your hands on some information which should look useful. Example:

2016-03-08 22:30:53,263 [8] DEBUG Zw.JsonLogViewer.AppBootstrapper - Creating bootstrapper
2016-03-08 22:30:53,717 [8] INFO  Zw.JsonLogViewer.ViewModels.ShellViewModel - Loading last-used logfile: 'D:\Daten\Tech Files\Development\dotNET\..\bin\Debug\1\lastrun.json.log'
2016-03-08 22:30:53,729 [11] DEBUG Zw.JsonLogViewer.ViewModels.LogViewModel - Opening log 'D:\Daten\Tech Files\Development\dotNET\..\bin\Debug\1\lastrun.json.log'
2016-03-08 22:30:53,737 [11] DEBUG Zw.JsonLogViewer.Parsing.Parser - Parsing log file 'D:\Daten\Tech Files\Development\dotNET\..\bin\Debug\1\lastrun.json.log'
2016-03-08 22:30:53,859 [11] DEBUG Zw.JsonLogViewer.Parsing.Parser - Parsed 331 lines into 331 log entries
2016-03-08 22:30:53,860 [11] DEBUG Zw.JsonLogViewer.Parsing.Parser - Providing 6 keys as columns: date,level,appname,logger,thread,message

Write done what is important in a nice human readable way. No binary dumps needed 😉

Okay, now that you got this 8MB of text right infront of you, the text editor for your choice will search for “ERROR”, a logger name and other subjects. Some advanced editors are able to hide all lines in a file which do not match your criteria. This way you can try to only show INFO messages.

When you have done this, you will find out that handling logs with several hundreds or even more lines can be a pain. Although these files are there to help you finding bugs – and might be your only debugging help in production scenarios (where it might be not possible to connect with a remote debugger).

Machine Parsable

Of course, I do not want to go back to any kind of proprietary log format – less readable and you have to write your own parser. A good idea is to go with a well know format. XML got infamous because most people do not like XML namespaces and the redundant information due to the closing tags. The fall of XML was the rise of JSON. It is a more lightweighted format (also there is an optional schema feature out there) and parsers are available for almost any programming language. If we provide the information about our log statements as JSON objects it could easily be parsed.

{"date":"2016-03-06T18:18:37.3018646+01:00","level":"INFO","logger":"State","thread":"9","message":"Beta 'Anzu' 2.2.1.3 starting"}
{"date":"2016-03-06T18:18:37.4539737+01:00","level":"DEBUG","logger":"MyApp.App","thread":"9","message":"This log channel is DEBUG enabled and might contain really much information. For production usage we recommend setting the log level to INFO or WARN."}
{"date":"2016-03-06T18:18:37.5140168+01:00","level":"INFO","logger":"MyApp.AppBootstrapper","thread":"9","message":"Creating bootstrapper..."}
{"date":"2016-03-06T18:18:37.5820649+01:00","level":"DEBUG","logger":"MyApp.AppBootstrapper","thread":"9","message":"Running bootstrapper configuration"}
{"date":"2016-03-06T18:18:37.7261679+01:00","level":"DEBUG","logger":"MyApp.SysCore.License","thread":"9","message":"Providing empty license without translation"}
{"date":"2016-03-06T18:18:37.7411796+01:00","level":"DEBUG","logger":"MyApp.App","thread":"9","message":"Application is starting..."}

As you can see this approach assumes every file line is a JSON object. You could also have the complete file to be an array of JSON objects – but this would require to rewrite the end of the file all the time to enlarge the array. If the app crashed and the log writer cannot flush, you might end up with a destoryed JSON array. Therefore I prefer the “one line = one JSON object” approach.

You can image, this is slightly harder to read for a human but not too hard. Also using a text editor’s search feature might work even better because of the object property names visible all the time.

A format like this can be achieved in log4net with the additional libraries log4net.Ext.Json, log4net.Ext.Json.Serializers.Newtonsoft and (of course) Json.NET (former Newtonsoft.JSON, no .NET app using JSON without it). This is a rolling file appender configuration example:

<appender name="RollingFileJson" type="log4net.Appender.RollingFileAppender">
  <file value="lastrun.json.log"/>
  <appendToFile value="false"/>
  <rollingStyle value="Size"/>
  <maxSizeRollBackups value="5"/>
  <maximumFileSize value="10MB"/>
  <layout type="log4net.Layout.SerializedLayout, log4net.Ext.Json">
    <decorator type="log4net.Ext.Json.Serializers.Newtonsoft.NewtonsoftDecorator, log4net.Ext.Json.Serializers.Newtonsoft" />
    <renderer type="log4net.ObjectRenderer.JsonObjectRenderer, log4net.Ext.Json">
      <factory type="log4net.Ext.Json.Serializers.Newtonsoft.NewtonsoftFactory, log4net.Ext.Json.Serializers.Newtonsoft" />
    </renderer>
    <default />
    <remove value="ndc" />
    <remove value="appname" />
  </layout>
</appender>

This is a lot of assemblies but you might use Json.NET anyways and the other two are due to the modularity. And you could still add your own renderer.

With all this great machine-readable output there is one question left.

How can I read it with filtering and sorting?

I was suprised when I did not find  a nice little tool which would parse any kind of JSON objects into a column based view. If you can recommend any tool for this I would highly appreciate any comment to this post!

What I was looking for was a tool where not all lines would require the same JSON object. There could be a property only added to specific log lines for example. The tool should not crash neither ignore an inconsist JSON object layout through the log file.

Zw.JsonLogViewer

Did I say it would be easy to parse log entries in JSON format? Yes, it is. So I built a small MVVM application to open those files and be able to search, filter and sort log entries.

zwjsonlogviewer

This small sample app is available at GitHub as open source under the Apache 2 license.

It currently features:

  • Full text search through all entries.
  • Providing columns for all object properties found through the complete file.
  • Sorting and filtering based on columns.
  • Detailed JSON view for the selected entry as well as detailed JSON object property view (seleccted through the columns’ context menu).
  • Auto-refresh on file updates.
  • Optional automatic re-opening of the last loaded file.
  • Hiding columns.

Known limitations:

  • The file will be completly loaded into memory so it will consume quite a lot of it.
  • No recent files feature.
  • Only one filter value per column, so you cannot filter on ERROR and WARN for example.

This is just a small basic application but it might come in handy. And you are welcome to send push requests 😉

Further Noteslogs

The concepts here apply not only to the .NET platform but also Java, Ruby, etc – programming in general. Also there a companies providing specialized logging services for desktop and mobile apps. If you are going big, have a look at those – especially when going mobile or cloud where you might require to collect and analyze data from thousands of devices per day or even minute.

Also a log: “a portion or length of the trunk or of a large limb of a felled tree.“

]]> https://blog.zoolway.com/logging-human-readable-vs-machine-parsable/feed 1