woensdag 11 augustus 2010

Web Service Task : "The selected Web method contains unsupported arguments"

Hi,

In a former post of mine i used a script task in SSIS for connecting to a webservice. When hovering the toolbox in SSIS i noticed a Web Service Task and i thought: "Why not try this for connecting to the Bing webservice for getting the geocode information". Okay...well that isn't that simple.  So i started to set up a HTTP connection manager. Something like this:



The next thing is to setup the Web Service Task:



Hey, what's that ? : "The selected Web method contains unsupported arguments.". Searching the internet, it appears to me that the Web Service Task is very.... very limited as i read on several blogs:
As Simon Sabin complains, there is no documentation about the unsupported features. I think that the Web Service Task is built on a certain WSDL version and newer versions are not supported (still in 2008 R2!).

I've seen some examples that do work:
So i've read a couple of posts of people who did find a solution by adjusting the WSDL file, by dropping or commenting out some elements that are beyond SSIS capabilities:
  • Deleting complextypes from WSDL File.
  • There's an issue with GUID
May be the best advice is: DON'T USE THE WEBSERVICE TASK AND USE THE SCRIPT TASK!
The book "Professional Microsoft SQL Server 2008 Integrations Services" tells me that the web service task is not a common source for large volumes of data and this is why Microsoft doesn't pay a lot of attention to this feature, off course.

Gr,
Hennie

donderdag 5 augustus 2010

What is data?

I follow now and then some discussions about data. Some say: “well data is like water; it flows through your organization and people will use the data as they like”. It’s a nice analogy but it’s too simple to say that, because data can be of all kinds of forms: simple, complex, float, text, etc. So data is flowing through your organization as little, big, roundy or roughly pieces. Little pieces for the easy businessprocesses and complex data for complex analytical questions. So, data can have all kind of forms and characteristics. Let’s describe some more characteristics of data:

• Data is a representation of the world around us, at some moment in time. Some data will be accurate (temperature readings) and some won’t (human entry systems). So, especially, don’t expect that data is non - erroneous representation of the world around us. It will allways has flaws. This is what we call data quality. How accurate describes data the world around us?

• Data has a life cycle. Data is created, updated, expired, renewed, destroyed, backupped, etc. In a datawarehouse we  mostly keep all the data for years in a detailed level. Why? Well, because we can, we never know if we gonna need it, we don’t know how we are gone use it (lowest grain), because diskspace is cheap, etc etc. All reasons to keep the data in your systems.

Data about fashion sales in a fashionshop of the last day can be very important. Fashion sales from a couple of months ago about a certain day will be less important. May be data should aggregate more when it's get older. Below you can see a couple of data life cycle curves positioned against importance.



• Data can be very valuable at a moment in time. This depends on the success of the actions of an organization, department or person. Data is only valuable when it’s used. If it’s not used it doesn’t represent a value.

• Data can be aggregated to a higher level or it could be kept at a low level (grain).

• Data describes something and it can describe things : master data and it can describe events: transactional data.

• Data will be less valuable when the source is difficult to determine or unclear. Also when the data is ‘scrambled’ during ETL processes users tend to discuss this and could feel unsecure whether the data is correct.

• Data could be objective or it could be subjective. Objective when it’s trivial, like the address of a customer but it will be less objective when the data is processed for business decisions. Mostly data quality is mentioned with operational data like an address of a customer but when the data is aggregated to a higher level the ‘data quality’ of this indicator is also important. In this case the business rules should be judged for correction adaptation.

Gr,
Hennie

dinsdag 3 augustus 2010

Geography (Part II)

In the first article about spatial information, we have seen an example of transforming some textual based location information into a longitude and latitude with a webservice. But even this information is not ready for using in a map. Therefore we need to convert the longitude and latitude into some sort of geometrical points.

SQL Server 2008 introduced the geometrical datatype, which gives some more flexibility and a better use for maps. As stated in an article on developerfusion, there is also another type Geometry. The SQL Server team has concluded that this type based on open standards wasn’t adequate for some scenarios. In the artivle Siddharth Mehta he introduced a script task in SSIS in which he updates the geocoded data with an UPDATE statement. I decided not to use this and decided to code this in T-SQL.

First i had to decode the , in . because i had my regional settings set on Dutch and the POINT expects . as an inputparameter. After that i used the static function geography::STPointFromText.

This gives the following results:


SSMS sees that we are working with a spatial column in our result set and displays an new tab : Spatial results:


At first i didn't see anything, after that i thought that i saw some dirt on the screen but when i hovered over the points a tooltip showed up.

Ok, next time we're gone built a report with report builder 3.0 and convert this information into some visual information.

So that's it for now!

Greetz,
Hennie

zondag 1 augustus 2010

System-of-record-only implementation

This is the third article in the series about MDM based on the article “Master Data Management from a technical perspective” from Microsoft (april 2010). As I mentioned in the first post, i have noticed that there are 4 implementation possibilities of Master Data management:

 
• Master data registry implementation
• Data aggregation implementation.
• System-of-record-only implementation.
• Complete enterprise master data management implementation.

 
This blog is about the 'System-of-record-only implementation’. In this implementation there are more systems of entry's in contrast with the data aggregation implementation. The data is transported bidirectional to the system of entry systems. This is solution which i had in mind when i started studying MDM. Data is entered in any systems and it's compared with MDM system.

 

 
 
These implementations still require a degree of data integration and ongoing cleansing as elements may come from both the source system and the master data management application. Also, many times this system of entry only has the ability to detect data issues directly related to the initial use. For instance, any customer information that is not stored in the CRM solution will not be available to determine complete data quality.
 
Advantages:
  • MDM can run in the background
  • Less impact for users.


Disadvantages
  • Complexity.
  • On going cleansing.


Greetz,
Hennie

dinsdag 27 juli 2010

Geocoding text based spatial data wih SSIS packages.

On SQL Server Performance site I found an interesting example of geocoding some spatial data of siddharth Mehta. In his example he decodes some famous places around the world into a latitude and a longitude. I decided to try this and to learn more about geocoding. Below you can see the result.

These are the steps you have to take for this small demo:
1. Create testdata.
2. Use geocode processing engine (Bing or Google).
3. Built a geocode webservice wrapper class.
4. Built a SSIS package
5. Write the results in an table.

1. Create the testdata

In the example of siddharth Mehta he used some objects from around the world. I decided to add some places and tried some tests with it. Below you can see the starting point i used.


2. Use a geocode processing engine

We need to use bing maps or google maps to process the information, as we provide in the example, which can take text based location information as an input and return the latitude and longitude of the corresponding location. These web based applications expose web-services which can be consumed by applications, our purpose in this case is to geocode the text based data we have in our "Address" table.
For the purposes of this sample application, we will use the webservice exposed by Bings Maps. First we will need to subscribe to the service. Navigate to www.bingmapsportal.com, sign up and create a key which will be your gateway to the webservices exposed by Bing Maps. In the below screenshot, you can see that I have created my application-key pair.


3. Built a geocode webservice wrapper class

As the article states, create a client or wrapperclass which can be used in the SSIS package. This way it’s easy to use a webservice. The .NET framework ships with the wsdl.exe tool. This tool creates a proxy class. Installing the .NET framework was not enough in my case.  Even when i installed the SDK for .NET framework icouldn't find the wsdl tool. I had to install visual studio express before the wsdl tool was installed. May be I missed something.

Do this:
wsdl /out:C:/geocode.cs http://dev.virtualearth.net/webservices/v1/geocodeservice/geocodeservice.svc?wsdl

This will create a geocoding class in C#. This will be used in the SSIS Spackage.


4. Built a SSIS package

The next step is building a SSIS package.There are four steps in this package:
a. Read the textual spatial information
b. Add two columns: Longitude and latitude
c. Retrieve the information with the proxy class and bing webservice
d. Create a destination.

a. Read the textual spatial information.
  • Create a SSIS package and give it a name.
  • Create an OLEDB connection to the Address table.
  • Add a Dataflow to the package and configure it to read the address table 
b. Add two columns: Longitude and latitude
  • Create a derived column task.
  • Add two fields latitude and longitude
  • Use the unicode textstring and set the length to 100



c. Retrieve the information with the proxy class and bing webservice.


  • add a script transformation task
  • Select Type transformation.
  • Select scripting language visual C#.
  • Edit the lattitude and logitude columns and make them writeable.
  • In the project explorer choose "add existing item".
  • Add references System.Web.Services and System.XML to the project.
  • Select the proxy class Geocode.cs which you created with WDSL tool.
  • Add the following code snippit in the Script task (Input0_ProcessInputRow)
{
string latitude = "", longitude = "";
GeocodeAddress(Row.AddressObject + ", " + Row.AddressCountry + ", " + Row.AddressCity, ref latitude, ref longitude);

Row.Latitude = latitude;
Row.Longitude = longitude;
}

private void GeocodeAddress(string address, ref string latitude, ref string longitude)
{

string key = @"xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx";

GeocodeRequest geocodeRequest = new GeocodeRequest();

// Set the credentials using a valid Bing Maps key

geocodeRequest.Credentials = new Credentials();
geocodeRequest.Credentials.ApplicationId = key;

// Set the full address query
geocodeRequest.Query = address;

// Set the options to only return high confidence results
ConfidenceFilter[] filters = new ConfidenceFilter[1];
filters[0] = new ConfidenceFilter();
filters[0].MinimumConfidence = Confidence.High;
// Add the filters to the options
GeocodeOptions geocodeOptions = new GeocodeOptions();
geocodeOptions.Filters = filters;
geocodeRequest.Options = geocodeOptions;

// Make the geocode request
GeocodeService geocodeService = new GeocodeService();
GeocodeResponse geocodeResponse = geocodeService.Geocode(geocodeRequest);

if (geocodeResponse.Results.Length > 0)
{

latitude = geocodeResponse.Results[0].Locations[0].Latitude.ToString();
longitude = geocodeResponse.Results[0].Locations[0].Longitude.ToString();

}
}

  • Replace xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx with the key which gave Bing in the BingMapsPortal.
  • Not sure if this is needed, but do a build.
d. Create a destination
  • Write the results in a table with Destination


5. Write the results in a table.


And here are the results:

Conclusion:

So this example shows the transformation of turning the textual spatial information into a spatial information. I tried some some small places and they all got a latitude and longitude. This way you can transform spatial information in geocoded information. Interesting and i will try more of this stuff and blog abou it.


Hennie