SMA – SMA Runbook Toolkit (SMART) Watch The Encoding

Recently while building some serious SMA runbooks I bumped in an issue. We usually develop the SMA runbooks in different editors like PowerShell ISE / SMA GUI / Notepad++ and keep them saved in TFS online, which we managed through Visual Studio. After we finished developing the runbook, which would update the runbook activity in Service Manager, I discovered some strange characters in the description field.

The description of the review activity looked like this…

Our development process works like this. First we write a runbook in our favorite editor and if we have a working release we copy the script to the SMA development environment (DEV) and see how it goes. If everything works as expected we use the SMART toolkit to export and import the runbooks into the integration environment (INT). This process worked great so far and we have been doing this for dozens of runbooks without an issue. At one point there was a need to write some German text in a description field of a review activity in Service Manager. In the screenshot above you can see that Service Manager could not display certain characters properly.

By checking the raw runbook in TFS everything seemed to be ok, it appeared like this…

The description text contains some summary data from the service request written in German language and some French character. The description text is wrapped in a here-string and the problematic character is an “à”. I was wondering how the script looked in SMA and therefore I checked the editor in SMA. The runbook itself in SMA looked like this, notice the “rectangle” characters…

The only difference between the TFS version and the SMA version was, that the SMA runbook had been imported using the SMART tool. This confirmed my suspicion that the import process could not deal with this special “à” character . I checked other places where I had some other “non-English” charaters like “ä,ö,ü etc.” and they did also not appear correctly. After some more investigation, I had another runbook which also contained such problematic characters BUT it did not create any problems in SMA nor in Service Manager. All characters appeared properly as they should.

Hmm…

Let’s summarize so far, I have two runbooks both contain special characters, both were imported through the SMART toolkit BUT one runbook did NOT display the special characters properly in SMA / SCSM and one runbook displayed the description string correctly.

Analysis:

Luckily we manage the runbooks in TFS so I made a copy of both runbooks and looked at the files in a HEX editor. Why? Well, if you think about it, runbooks are just PowerShell files and PowerShell files are just plain text files with a “PS1” suffix. To see if there is a difference we need to analyze the text file in a HEX editor “to see behind the curtain”. A possible choice since we use Notepad++ a lot, is to install the HEX Editor plugin for Notepad++…

I resarted Notepad++ then I opened both files and switched to HEX view. On the left side the “OK” runbook and on the right side the “Not OK” runbook which caused some problems.  We clearly see there is a difference in the code…

Just from a first brief look we can tell that both files are not the same “format”. The “format” or better character encoding of the file seems to be different. Therefore let’s check both encoding types in Notepad++…

In the example above we also see that the “OK” script uses a UCS-2 Little Endian (UTF-16) encoding and the problematic “Not OK” file an ANSI encoding. By just looking at the files itself we cannot recognize any difference. Well I am not an encoding expert, but as far I know the most popular encoding used in the World Wide Web area is UTF-8 and also if I look at the metadata of the SMA web service https://smaserver:9090/00000000-0000-0000-0000-000000000000/$metadata the encoding type used in the header is UTF-8.

Conclusion:

If we look a the SMART toolkit it uses the Import-SmaRunbook cmdlet which finally uses the SMA web service to import / update the runbooks. This leads us to the conclusion, that it is best to have all your runbooks you write in a UTF-8 encoding format. Usually you will not encounter any character problems as long you just write plain PowerShell, because there you don’t have any special characters. BUT(!) as soon you start adding free text in your PowerShell workflows within a here-string or in a regular string written in your native language which might be German, Dutch or any other language  you are very likely to run into this issue when you try to import your runbooks using the SMART toolkit or the Import-SmaRunbook cmdlet itself.

Solution:

How can you avoid running into this issue? Well as I mentioned you need to have saved your scripts as UTF-8 encoded. If you use PowerShell ISE in version 3 and higher ALL your scripts are natively saved as UTF-8. In this screenshot I just checked my PowerShell ISE on a Windows 8.1…

As soon you are editing / saving your scripts in different editors or your are copying your files around from one environment to another it could happen that this changes your encoding of the file. Is there an easy solution to fix this problem?

Well, if you just have one or two files you can check the encoding in Notepad++ or any other capable editor.

If you have multiple files you could use this PowerShell script to check the encoding. Finally if you really just want to convert everything in bulk to UTF-8 in one shot, I provide here a simple script that just uses a source directory and a target directory and will the copy and convert all files from the source directory recursively to the target directory in the UTF-8 format. I converted the problematic files in my environment and everything appeared as expected.

You can find the script here, I hope this helps.