Month: January 2014

Learning from my mistakes on SQL…

Here’s some design mistakes I’ve made in the past… learn from my example and shun these anti-patterns!

  1. Normalization is here for a reason. Normalization defines a set of methods to break down tables to their constituent parts until each table represents one and only one “thing”, and its columns serve to fully describe only the one “thing” that the table represents. If you find yourself using string operators in WHERE clauses a lot (SUBSTRING, CHARINDEX, LIKE) to parse out a value in a column – you’ve missed the boat design-wise and your data will become less and less searchable as you move away from the SQL paradigm. (And don’t get me started on COLDESC1, cOLDESC2, etc)
  2. Natural keys are the thing, man. Think carefully before you use an identity or a GUID as your only key. You should reconsider your design if you can’t pick out which row you want from a table without knowledge of the surrogate key. Each of your tables in general should have a natural key that means something to the user and uniquely identifies each row in the table – i.e. go with PartNumber over an integer PartID GUID. In the extremely rare event that there’s no natural key (like with an audit table) – then and only then use a surrogate key.
  3. Protect your data, dammit. Your tables should have constraints on nullability, string length, FK assignment, etc.
  4. Naming standards are actually important. For example, think about X304E_DESCR. It’s important to be CONSISTENT above all and be descriptive – this is your first line of defense when it comes to documentation. Make your names clear, simple and descriptive – and don’t include metadata in the column names.
  5. Don’t use one table to hold all your domain values. Remember that SQL works in sets, according to normalization rules. If you’re doing joins like “AND customer.TableName = ‘Customer’ and customer.RelatedToColumn = ‘descr’, well jeez… that’s a performance killer and will quickly make your queries maddeningly hard to write and update. Remember, tables should represent one ‘thing’ – and one thing only.
  6. Avoid dynamic SQL unless there’s truly no other choice. The same reason above applies to trying to code generic T-SQL objects – such as accepting a table or set of column values as a parameter. It seems neat and tidy to do this initially as it can handle multiple tables with one sproc – but we don’t think of the performance implications of misusing sp_executeSQL in this way. Sometimes there’s no shortcut to writing CRUD statements against the entities directly – or using EF.

Looking through this article on the interwebs, I’m thinking a lot of my experience here isn’t unique. Tip of the hat to that blogger and the great books from Celko and the Guru series from Ken Henderson.

ETL from hell – getting past denial

A great article here on diagnosing problems with batch system performance: https://www.simple-talk.com/sql/performance/the-etl-from-hell—diagnosing-batch-system-performance-issues/

Basically it goes like this – when there’s problems with batch processes overlapping or timing out, companies typically try the following:

  1. Can we ‘fix’ this with more hardware? (this is expensive, and a temporary fix)
  2. Can we change the product – for example from SQL to Oracle? (this only work with a redesign. Usually the product isn’t the issue!)
  3. Turnkey solution (expensive and typically doesn’t address the root issue)
  4. Split systems into parts, each on isolated servers (hard to synchronize running jobs across your now-separate systems)

Instead of these knee-jerk reactions, there’s a better way – both strategic and tactical – to fix this:

  • Tactical
    • Gather statistics – how long is a process likely to take?
    • Get a baseline of long-running problem queries
  • Strategic
    • Determine processing time windows
    • Create a task list in Excel – and include internal/external dependencies
    • Monitor the system – and include management in monitoring reports
    • Define threads – tasks in sequence – and processing times
    • Implement batch system control that runs off your table of tasks/dependencies

Conduit… to HECK!!!

So, I did something foolish the other day and downloaded a virus. (Long story short, and it’s no excuse, I searched for “filezilla” and then clicked on the first link I found – without checking where it was pointing to. Next thing you know, every time I opened a browser I was looking at this “Conduit” helpful search site… with all kinds of malware sniffers no doubt spinning away happily collecting my credit card information. (Like those poor companies haven’t suffered enough). Worst part is, I couldn’t do what I usually do – uninstall the program and reset any search engine settings – Windows Uninstaller wouldn’t remove it. And, I didn’t want to do a registry hack. What to do?

  1. Removed Conduit from IE and uninstalled Google Chrome.
  2. Installed CCleaner and ran it. (This didn’t remove it unfortunately)
  3. Install AdwCleaner and ran (this got rid of the bulk of the settings
  4. Installed Malwarebytes and ran (found some more settings
  5. reinstall Chrome.

And that’s it, I’m “clean” now. I feel embarrassed and uncomfortable now, EXACTLY as if I’d caught some kind of terrible disease from being where I shouldn’t be. Let that be a lesson to myself!

WebAPI – the new WCF data services. And KnockoutJs/MVVM explorations.

Here’s a link to the TryOutMVVM project. Enjoy!

There’s a ton of links at the bottom of the page, but in terms of best explanation/clearest walkthrough it’s hard to beat this one as a template. Skip past all the Azure propaganda – about halfway down is where the good stuff is at.

WebAPI is super cool and allows us to do things that the “old” WCF data services only dreamed of – such as returning nested (hierarchical) tables, etc. These days I find it difficult for us to justify building a data layer on anything else. The thrill of being set free from procedural logic/sprocs, I can’t even express!

Here’s a sample return from a WebAPI. (JSON could be set up just as easily.) What’s involved in getting this set up? And can we mess around with linking it up to a web frontend and, I don’t know, sprinkle on some KnockoutJS for asynchronous goodness?

Let’s start with a few words on MVVM versus MVC

In terms of design MVVM is a logical extension of MVC. Model-View-Controller works off abstraction, where the model layer is unaware – completely uncoupled – of the view layer. In MVC, the Controller does all the work for this – and is responsible for determining which View is displayed in response to any action including when the application loads.

MVVM in contrast uses the ViewModel layer as a concept. It includes three key parts:

  1. Model (Business rule, data access, model classes)
  2. View (User interface)
  3. ViewModel (Agent between view and model)

Here the ViewModel acts as an interface and …

  • provides data binding between View and model data.
  • handles all UI actions by using commands.

In MVVM, the ViewModel does not need a reference to a view. The view binds its control value to properties on a ViewModel, which, in turn, exposes data contained in model objects.

Let’s walk through what it will take to implement a one-table MVVM app:

  1. Create a new MVC project, and enable both Webforms, MVC and Web API. Call it “ContactManager”.
  2. Modify _Layout.cshtml with contents (see zip file)
  3. Add a Contacts.cs class in Models folder, and Ctrl-F5 to build. EF uses reflection so these periodic builds will be necessary.
  4. Add a controller. Select MVC Controller with R/W actions Using EF. Call it HomeController; use the Contact model class, and select a <New data context>.
  5. In the Tools menu, open up Package Manager Console. At the prompt enter:
    1. enable-migrations (since we have multiple contexts, enablemigrations -ContextTypeName
      ContactManagerContext)

    2. add-migration-initial
  6. Open up Migrations\configuration.cs, and add the code in the zip file. (this fills in the seed method to populate the database)
  7. Back to Package Manager Console, enter: update-database
  8. Build.
  9. Replace index.cshtml
  10. Add a Contacts.css style sheet to the Content folder.
  11. Add a line to the App_Start\Bundle.config to register knockout.
    1. bundles.Add(new ScriptBundle(“~/bundles/knockout”).Include(“~/Scripts/knockout-{version}.js”));
    2. bundles.Add(new ScriptBundle(“~/bundles/knockout”).Include(“~/Scripts/knockout-{version}.js”));
  12. Add a Web API Controller with R/W actions Using EF. Call it ContactsController. Select Contact for the model, ContactManager context.
  13. Now build it, Ctrl-F5. Check out the new Home controller, at {yourappurl}/api/Contact.

That’s just a quick overview. In more detail, let’s go thru this step by step:

  1. In Visual Studio, create a new Web Application. Call it “TryOutMVVM”, and enable MVC, Webforms and WebAPI in the setup (ignore the test project, we won’t use it here.)
  2. Modify _Layout.cshtml with contents (see zip file)
  3. Add a TryOut.cs class in Models folder, and Ctrl-F5 to build. EF uses reflection so these periodic builds will be necessary.

    See above. This represents a single entity – a ID field, a product and location link (which in an actual application would probably be a related table with a foreign key), and some information about this try out. In this case, this represents a Line Try Out – an engineering change to a production line, including what part numbers are impacted, the person making the change and the description, the location and product it links to, and an identifying #.

  4. Add a controller. Select MVC Controller with R/W actions Using EF. Call it HomeController; use the Contact model class, and select a <New data context>. Note, I got stuck on this a few times and couldn’t figure out what I was doing wrong – the code built, but I was looking at a blank response in Fiddler. Turns out I was selecting the wrong type of controller (kind of like Homer Simpson adjusting Marge’s camera with a drill and hammer. And, my mind just got distracted and a quick youtube search of that episode reminds me of how awesome that show really was back in the day. Ah, sorry, back to reality…) Anyway, and I can’t stress this enough, make sure you select the RIGHT kind of controller for this and the following steps!!! If you get an error message here it’s usually because you haven’t built the project yet (see #3 above) – Ctrl-F5 and THEN add the controller silly! I also usually delete the HomeController first, but go ahead and overwrite if you forget.

  5. In the Tools menu, open up Package Manager Console. At the prompt enter:
    1. enable-migrations (since we have multiple contexts, enablemigrations -ContextTypeName
      ContactManagerContext)

    2. add-migration-initial
  6. Open up Migrations\configuration.cs, and add the code in the zip file. (this fills in the seed method to populate the database)
  7. Back to Package Manager Console, enter: update-database
    1. Here’s a sample of what it will look like in PMC. Check out the Migrations folder for a sample of what will be created:

      And, as long as we’re here, check out Views\Home\Index.cshtml file. Notice all the work that EF handled for us in scaffolding actions:

  8. Build, Ctrl-F5. OHMYGAWDDIDTHISJUSTHAPPEN?!!!?!!

  1. Replace index.cshtml with what’s in the ZIP file.
    1. Some comments. Notice this is plain and simple HTML5/CSS3. Very clean, really, for all it does. In a future version you’d want to move all those javascript functions out to a separate .js script. But this does work:

  1. Add a TryOut.css style sheet to the Content folder, and fill it with the styles in the ZIP. Again, nothing fancy here.
  2. Right click on the project and select Manage NuGet Packages. Search online and add a reference to knockoutjs. Now, look at the Scripts folder and see which version NuGet dropped in there – mine is 3.0.0. Make a note of this, you’ll need it for the next step.
  3. Open up App_Start\Bundle.config.cs and add the following lines:

    bundles.Add(new StyleBundle(“~/Content/css”).Include(

    “~/Content/bootstrap.css”,

    “~/Content/tryout.css”,

    “~/Content/site.css”));

    bundles.Add(new ScriptBundle(“~/bundles/knockout”).Include(

    “~/Scripts/knockout-3.0.0.js”));

  4. We’re actually almost there, can you believe it! Let’s add a service layer using WebAPI. Right click on Controllers and add a Web API Controller with R/W actions Using EF. Call it TryOutController. Select TryOut for the model, TryOutMVVMManager context.

    note, there’s no Views created for this – since its web aPI. All the API calls are right there in our TryOutController.cs method headers.

All that’s left is the building and the screaming. Let’s build it, Ctrl-F5.

Check out the new Home controller, at {yourappurl}/api/Contact.

Have some fun with it. Try

http://localhost:58149/api/Tryout

http://localhost:58149/api/Tryout/1

(where the port # obviously is your URL). Wow!

Notes and Links

I’m not done yet. I want to do some more research about the Repository pattern and how to do Web API the RIGHT way. I think I need to spend more time thinking about the design pattern and get out of the database-first box.

Next, for the frontend, I want to start thinking about KnockoutJS and the MVVM model:

f

The single best example – used as a base for this posting: https://www.windowsazure.com/en-us/develop/net/tutorials/rest-service-using-web-api/#bkmk_createmvc4app

A long post based on a single example, with CRUD operations. I like this ahead of the ones below, for ease of use. http://www.asp.net/web-api/overview/creating-web-apis/creating-a-web-api-that-supports-crud-operations

A good walkthrough to get started: http://www.asp.net/mvc/tutorials/getting-started-with-ef-using-mvc/creating-an-entity-framework-data-model-for-an-asp-net-mvc-application – a student-enrollment-course many to many type relationship sample.

http://www.asp.net/web-api/overview/creating-web-apis/using-web-api-with-entity-framework/using-web-api-with-entity-framework,-part-1 – this is by Mike Wasson, who created the excellent Movies web api sample. This is more of a master/detail page view creation. It’s excellent because it also goes through KnockoutJS, and has a section on using Web API with webforms.

I need to add user authentication and roles – this article contains some great info on hooking up users/roles.

Jason Zander’s blog post – has a nice walkthrough on the Web API: http://blogs.msdn.com/b/jasonz/archive/2012/07/23/my-favorite-features-entity-framework-code-first-and-asp-net-web-api.aspx

Best example I’ve seen yet on Web API – including the otherwise excellent MVC book by Apress – is this post: http://www.dotnetglobe.com/2012/03/crud-operation-using-aspnet-web-api-in.html

See the MSDN article here on Single Page Applications and MVC vs MVVM.

Another article on Web API and how it could be used vs WCF Data Services – http://msdn.microsoft.com/en-us/magazine/dn201742.aspx


Noodling around with Entity Framework.

Entity Framework is the default way that Microsoft wants us to be noodling around with data. And it’s easy to see why – this is MSFT’s answer to the challenge posed by Ruby, well over ten years ago. EF gets better with every version.

 

 

From the Getting Started documentation, it should just be a simple matter of

  1. Add a new diagram – and make sure for delete (for example) all changes are set to cascade.
  2. Add a new web form + master page.
  3. Add an Entity Data Source. In design view, configure it to point to your Entities data source you created in step #1.
  4. Add a new GridView. Select your entity data source. Select Enable Paging…. down to Enable Delete. Delete the ID fields.
  5. Set the Dates template, etc on any custom fields.
  6. Set up the ContextTypeName attribute to enable lazy loading.

However, I’m getting a “The provider did not return a ProviderManifest instance” error message. This usually indicates an issue with the connection string. I’ve tried switching to Integrated Security, changing the username/pwd – no dice. I honestly think this is a bug with EF6, since I’m using the most recent version of Entity Framework and I can’t recall seeing this issue in EF5.

Since the forms use sprocs by and large, and we are using webforms, I’m OK with – for now – going old-school and using sprocs not EF for my data connections. For our new app though I am going to revisit this and use either Linq-to-SQL or preferably that nifty BackboneJS/KnockoutJS + MVC stack and a webapi data layer.