Adi's space - Windows Live

Adi's profileAdi's spacePhotosBlogSkyDrive

Tools

Help

Adi Saric

Occupation

Systems Architect

Location

Seattle, WA

Interests

parallel/distributed systems messaging computer vision cryptography quantitative finance

I am a University of Washington graduate and a 12-year veteran in software and systems development and integration. I especially enjoy working in financial/investment industry due to the nature/challenge of business requirements. Majority of my technical work and research is based on Microsoft technologies.

My big pet peeve is computers sitting idle and not doing any work. For that reason I try to design distributed systems that strive to most efficiently consume all of the available processor cycles.

"Things should be made as simple as possible, but no simpler."
--Albert Einstein

View profile details

Blog

March 27

TechEd 2008 Gadget: development lessons learned

In my previous post on TechEd 2008 Countdown implementation I promised to share development lessons/tips learned and this post serves that exact purpose.

Gadget development turned out to be most development fun I have had in a while. Even though it is fairly straightforward there are several exceptions to the standard Web page development that I would like to share in order to save you research time in the future:

Make sure to use both pairs of tags when specifying the location of your script file. A self-terminating tag such as <script.../> will not cut it and your gadget will not function as expected.
OnMouseOver event seems to be disabled as far as opening of the flyout window goes. The onClick event will have to do. If somebody has a solution for this one please let me know as I hate to disappoint my friends who requested this feature.
I was not able to set the background color of the gadget pages via bgColor property or the background-color style in a CSS file. However, the color specified in the style embedded in the BODY tag worked fine.
Opening flyout window with another one already open creates this weird "loss of focus" effect. In order to avoid this check if the flyout window is already open and if so simply refresh the information contained within it. Code below demonstrates this much better than my narrative.

if(System.Gadget.Flyout.show)
     //flyout already open - call its function that loads the data
     System.Gadget.Flyout.document.parentWindow.loadPost();
else
     //display the flyout
     System.Gadget.Flyout.show = true;

The time spent researching these items is nothing compared to the joy of seeing the final result in action (as shown below) and reading all of the great posts by the TechEd community members. The final product can be downloaded here or by clicking any one of the images below. Enjoy!

March 06

SQL Server 2008 to the rescue: user defined aggregate solution recovered

In my introductory post on user defined aggregates I mentioned that Merge() method will be an interesting point of discussion. The Merge() method is called any time SQL decides to split up the workload, process it on separate threads, and eventually merge all of the results. It seems perfectly desirable to be able to maximize the use of computer resources.

We have also decided to create a user defined aggregate that calculates the median which means that we will need some kind of data structure to hold all of the values in a group. For the sake of simplicity we could use an array of type double. The implementation using an array of type double will compile fine but upon the deployment to the instance of SQL Server you might receive one of the following two error messages depending on the data structure used.

Type {...} is marked for native serialization, but field {...} of type {...} is of type {...} which is a non-value type. Native serialization types can only have fields of blittable types. If you wish to have a field of any other type, consider using different kind of serialization format, such as User Defined Serialization.

Type {...} is marked for native serialization, but field {...} of type {...} is not valid for native serialization.

SQL Server has to be able to serialize UDAs in order to communicate/pass them between the threads. In fact, native serialization only supports following data types for native serialization: bool, byte, sbyte, short, ushort, int, uint, long, ulong, float, double, SqlByte, SqlInt16, SqlInt32, SqlInt64, SqlDateTime, SqlSingle, SqlDouble, SqlMoney, SqlBoolean. Since our array type is not acceptable for native serialization we will need to mark the Format attribute to use UserDefined serialization and provide our own implementation of Read() and Write() methods as defined in the IBinarySerialize interface.

The next issue that comes into play is that the maximum size of a serialized object cannot exceed 8000 bytes in SQL 2005. That does not allow too many double type values to be aggregated and the UDA execution will throw the following error if the size of serialized UDA exceeds 8000 bytes.

A .NET Framework error occurred during execution of user-defined routine or aggregate {...}: System.Data.SqlTypes.SqlTypeException: The buffer is insufficient. Read or write operation failed.

One suggestion I found was to force SQL to use on processor/core at the time via OPTION (MAXDOP 1) directive. That did not seem to resolve the issue as SQL Server does not seem to always accept aforementioned query hint. Needless to say this limitation greatly reduced the effectiveness of my solution.

Luckily the good folks working on SQL Server 2008 have decided to solve my problem and increase the size of the serialization buffer. SQL Server 2008 now allows CLR components to be serialized up to whooping 2GB as outlined in the TechNet BOL article titled "Requirements for CLR User-Defined Aggregates". Way to go SQL Server team and thanks for saving my project.

March 05

SQL Server User Defined Aggregates

Statistical calculations are quite important in the implementation of financial applications. Surprisingly SQL Server does not support many unless the process is running within the Analysis Services framework. Few months ago I was in a need of calculating a median in several stored procedures performing statistical risk calculations. While I could have used SQL approach like shown in the listing 1 below I would have to replicate the same piece of code in numerous stored procedures.

Listing 1: Median using T-SQL and SQL 2005 CTE, ranking

WITH data AS
(
    SELECT someID, someValue,
    ROW_NUMBER() OVER(PARTITION BY someID ORDER BY someValue) AS RowNum,
    COUNT(*) OVER(PARTITION BY someID) AS NumRows
    FROM dbo.someTable
)
SELECT someID, AVG(someValue) AS Median
FROM data
WHERE RowNum IN ((NumRows + 1) / 2, (NumRows + 2) / 2)
GROUP BY someID;

Instead I decided to write my own median function using CLR that can be used similar to the AVG built-in aggregate. Visual Studio makes things easy by providing a convenient template to get you started. In order to create an aggregate CLR function you first create a SQL Server Project and then add a new item/class based on the Aggregate template as shown below.

There are four methods listed below that the aggregate requires. The Init() method is called once per group and we can use it to instantiate our own variables, such as an array holding all of the values we are looking to find a median for. The Accumulate() method is called on every value addition of the group (think grouped records in a table) and Terminate() returns the aggregate value once the group aggregation completes. The Merge() method is used by the SQL engine itself which might decide to split the aggregation work to multiple threads and merge the results of which prior to executing Terminate() and returning the result.

public void Init()
public void Accumulate(double Value)
public void Merge(Median Group)
public double Terminate()

The Merge() method is especially interesting and will lead to further discussion in part 2 of this blog. If you are interested in my own implementation of percentile/median function please contact me via "Send a message" on my Spaces page and I would be happy to share my experiences with you in regards to this or any other user defined function.

February 20

My first Vista gadget: TechEd 2008 Countdown

gadget (definition courtesy of Merriam Webster)

Pronunciation: \ˈga-jət\
Function: noun
Etymology: origin unknown
Date: 1886

: an often small mechanical or electronic device with a practical use but often thought of as a novelty

My friends over at InDepth Technology approached me with a fun project recently. It involved creating a Vista gadget to act as a countdown to the largest technical event of the year - Tech·Ed 2008. I have never created a gadget before but the potential of making this a fun and exciting project made me jump right into it.

Interestingly enough creating a gadget is a very straightforward process. Technically it is nothing more than a combination of HTML/XHTML, CSS, and image files accompanied by a manifest in a form of a XML file. My gadget was simple enough that I decided to embed the JavaScript directly into the HTML file representing the timer counting down the seconds to the Tech·Ed 2008. Interesting point I have discovered in calculating time difference between now and the event start: if you express the event start with the UTC offset (in this case "Jun 3 2008 7:00:00 AM UTC-5:00" since the event is on the east coast) the subtraction of now and event date objects, event.getTime() - now.getTime(), will be properly adjusted based on the local machine time zone.

Here are the steps to create a gadget:

Create an HTML page providing a presentation layer and referencing relative path image(s) and CSS file(s) as necessary.
In the same folder or subfolder add your images and/or css/js files referenced by the HTML.
Add the gadget.xml manifest file describing the icon Vista should display when choosing gadget, copyright information as well as what HTML page will be used on gadget startup (gadget can contain multiple pages just like any Internet site).
Package all the files and subfolders into a zip file and rename the extension to gadget.

There is much more to gadget creation including the ability to access its own namespace for property/setting persistence as well as special presentation elements. I hope this post will be sufficiently interesting to get you started on your own gadget at which point you will need a bit more research into the advanced gadget functionality. In a meantime you can enjoy my Tech·Ed 2008 countdown gadget here.