The Death of the Hard Drive - and the Database Server ?

I'll start this post with a prediction: Hard drives are doomed

Actually, not much to argue with there for the informed observer. The new solid state hard drives can run rings around the mechanical version. Oh, they're not quite there yet - they're smaller and more expensive - but ten years from now you can bet that will all have changed.
And solid state hard drives are actually just memory - the only reason that they are packaged as hard drives is because that is the paradigm that we are locked into.

They're not the same memory as we buy for our motherboards.  That's dynamic, or volatile, RAM, where the data disappears with the power. The drives use static, or nonvolatile, memory (otherwise known as EEPROM).  The write cycles are orders of magnitude longer, and there is a limit to the number of write cycles per bit/byte/unit of storage, so the solid state drives have management units to spread the writing load around so as not to exhaust this limit.
Managing the write cycle limit is the one compelling reason to keep this memory as an independent unit, but I suspect that soon someone will find away around this limitation and also a way to greatly decrease the write cycle time.
And it's just a matter of time before the capacity increases beyond the ability of mechanics to compete with.

Once these things change, there will be no reason not to connect static memory direct to the processor. Why pipe it thought a slow interface when a 64 bit processor can directly address 18 million terabytes ?
The static memory will replace the role of the hard drive and probably some uses of the dynamic memory, and the dynamic memory will remain the fast, volatile work area it is now.

But the IDE/SATA/SCSI-connected box we call a hard drive will be gone.

And this brings us to the corollary of the first prediction:
Database servers as we know them are doomed

And that's because database servers are designed specifically to take memory based structures and store them in detached persistent storage - ie. a hard drive.

Now let's backtrack a fraction.
Database servers perform a lot of valuable and complex tasks, including not just storing the data, but also navigating it and keeping it consistent - transactions and query optimisation being two great examples.
So there's no way I'm predicting that these functions are going to disappear.

ACID is mandatory for any reliable system and in fact, I think this is a great litmus test for how any database system performs: assuming competent database design and indexing, can it handle an ad-hoc, highly complex query on millions of rows and still return results with low latency and throughput times ?
In the case of MS SQL Server and Oracle, the answer is yes (and server clustering is another game altogether, we won't cover it here). But many other database systems fail this test dismally.

But I do strongly believe that there will be a paradigm shift in how the data is actually stored. What concerns me is that none of the standard vendors appear to be gearing up for this.
Most of the serous RDBMSs utilise modelling of the hard disk topology at a very low level, so as to squeeze the maximum performance out of the disk.

So what happens when the hard drive disappears ? Well, the data will be stored persistently  in-place in memory. But we'll still need transactions. We'll still need a query optimiser.
And we'll still need to house the data store on a separate server and communicate with it somehow, whether by API or by SQL.
How is that going to look, and who's planning for this ? As far as I can see, as the Johnny Cash song goes: nobody.

Review of .NET Licensing Solutions

In preparation for my new application FlexMerge which will go on sale soon (plug: at www.arrow-of-time.com !), I've been investigating Licensing/code protection issues.
This is a windows class library which needs to implement serial number and license based protection using an activation server.
I also wanted the ability to automatically send serials from my ASP .NET website immediately on successful checkout.
I scoured the internet, coming up with a whole bunch of likely candidates:

DeployLX (by Xheo) ($700 +)
CryptoLicensing Professional ($300)
Infralution Licensing System ($170 + $170 license tracker source code)
Eziriz Intellilock ($179)
LomaCons InstallKey ($86 source code edition)
LicenseSpot (subscription)
Soraco Quick License Manager ($400)
Manco .NET Licensing System ($200/$480)

To cut to the chase, I ended up buying CryptoLicensing, as it offered equivalent features to any of the other offerings, along with samples of the serial generator API and an activation server, at a very reasonable price.
I bought it bundled with CryptoObfuscator.
I haven't rolled it out yet: watch this space. But in testing it has performed flawlessly.

Immediately discarded was LicenseSpot, as it offers a hosted, monthly subscription based service which looks great for those looking for a simple, hosted solution, but I was pretty sure the integration with my site was going to be a problem. I'd really rather have everything stored in my own database rather than spread over the internet.

Soraco Quick License Manager was also discarded quickly as its feature set appeared to be smaller than most of the other standard offerings. It didn't appear at first sight to support serial numbers, and I didn't have the time or inclination to go looking further.

Eziriz was an early frontrunner, but then I did run across reports of bugs, particularly with obfuscation, and tardy tech support. That resulted in a knockout too.
Note that price is not much of an object for me - with my hourly rate, it was going to cost WAY more to integrate the software than to buy it.

LomaCons InstallKey looked good, and is so cheap that I bought it to look at the source code rather than stuff around with a demo version. However, there were several bugs in the project which cost me time, and it  required IIS Express to work, which I didn't want to install on my Dev machine. In the end I couldn't get it to run, even WITH source.
It had a web based license manager with source code, but this was all written in old school ASP .NET WebForms style, which I have to say makes the gore rise in my throat. I wanted something I could tweak easily, and I would have ended up rebuilding that from scratch.

DeployLX makes a big effort to make their tools user friendly. But this comes with a price tag. This was easily the most expensive option. Since time is money, I was tempted to go with them for a bit, and benefit from a bit of hand holding, but it turned out that the features I wanted would invoke several more license charges over the base charge, and also that their serial generation API has certain license restrictions.
I think they would have worked, but I thought that he licensing restrictions were a bit onerous considering the other offers out here.

Infralution looks like a good solution. Also, pay a bit more for the source code. This one looked comparable to the Crypto product, but had a Winforms based license manager. Since I'll definitely be managing licenses online, this put me off a bit. All their ancillary tools dovetail into this license manager.
That said, they supply source, so rolling your own web based manager would have been doable and supported. I would have chosen these guys had Crypto not been around.

The Roundup 

All in all, there were several good players, and a lot of choice. It happens that the CryptoLicensing package was well reviewed, had a complete feature set, and also pairs with CryptoObfuscator which also is a plus.
Where I read of problems with licensing tools in general, it was often during obfuscation that this occurred, so buying a matched license/obfuscation pair (Crypto or DeployLX) makes good sense.

I'd highlight that I think most of these packages perform well for what they do, but being a developer myself, I have fairly demanding requirements.
Also, if you are REALLY short of cash, there are some very cheap solutions out there which I think would perform quite satisfactorily if you were willing to put a bit more time into making them perform.

I think the LomaCons comment that sometimes 'less is more' hits the nail on the head. In the end, I wasn't interested in the fluff of the 'all in one' integrated solutions, I just wanted a good solid bare-metal framework I could use for my own purposes. This meant a good feature set and an easy to use API.

Using USB devices with Oracle VirtualBox

I've had a great experience with Oracle VirtualBox. But like all software it's not perfect. There are two problems I've experienced with it.  Here is how to fix them.

(1) Error messages when trying to start the VM after upgrading

This is something that all the VM companies seem to have trouble getting right - I had the same problem with Parallels. When you upgrade, the VM config file format changes and and your old config file may not be successfully upgraded to the new version - it becomes corrupted. There are a variety of errors you can receive.
Just create a new VM image from the start, and configure it to the the same as the old one. It can be a bit laborious, but it does ensure a stable VM.

(2) Trouble attaching USB devices (eg. memory sticks) to the VM

Detection of the specs of a particular USB device is a bit tenuous.

The secret is:
- download and install the VirtualBox Extension Pack matching your installed VirtualBox version
- set USB2 active
- create a 'filter'

 - remove ALL details from the filter record apart from the description

This should detect anything that is plugged in.
If you do want to filter for a particular device, I'd advise writing down the specs that it comes up with initially on connecting the device, then clearing them as above and adding back one by one.

Note also that if you want to switch off detection of USB devices altogether, just remove the tick from the profile/filter. That way you keep the profile for turning on again later.

Reinstalling a Tracemaster Server

I've been doing support recently for a few customers whose servers have died.
Reinstalling Tracemaster can be quite an extensive process, especially when using scanners.
The following provides a list of resources and some pointers to help the process along.

Resources:

Install the latest SQL Server Express (at present, SQL Server Express 2008 R2 SP1)

http://low-bandwidth.blogspot.com/2011/10/installing-mssql-2008-express.html

Install the scanner software:
www.etracemaster.com/wlstudio_37.exe (Wavelink Server)
www.etracemaster.com/HTM_PDE_221a.exe (latest Tracemaster scanner program updates)

Install Tracemaster from the install CD.

Install the latest version of Tracemaster Software:
www.etracemaster.com/HTMUpdate_Packer_4_223f.exe (for Windows 7/XP)

Wavelink Setup:

(1) install wavelink server
(2) go to Start > Programs > Wavelink Studio > Wavelink Administrator
(3) got to Utilities > Authorization and copy the registration information over exactly from the old computer (check both Wavelink Studio NT/95, Wavelink Studio User Pack), or if the old computer is dead you will need the Registration codes which should be recorded elsewhere

(4) click the leftmost icon in the toolbar and start the monitor for 'localhost'
(5) Double click port 2001, remove the tick from 'Initialise Monitor on server startup', and click 'Shutdown'
(6) Click the second-to-left (green) icon in the toolbar to start up a new monitor. Choose TCP/IP, port 1000, in the next screen click 'browse' and select 'C:\Program Files\Harvestmaster\HTM_PDE_Stock_Control.exe'.
Tick 'Reset Monitor upon server startup', then leave the settings as-is and click 'Finish' in the final screen.

(7) Do the same again, this time using port 2000 and selecting 'C:\Program Files\Harvestmaster\HTM_PDE_PalletPack_IntLabel.exe' or one of the other pallet packer products depending on your packing process.

This completes the Wavelink setup. Wavelink and the two monitors should automatically start up when the computer is started.

If you open Wavelink Administrator and monitor localhost, it shows connections to the server as they happen, which can be useful for troubleshooting.

If you need to test that the Wavelink server is configured correctly (particularly when wireless scanners are not working), you can use the 'Virtual Client' in Start > Programs > Wavelink Studio > Wavelink Virtual Client.

Click 'configure' and create an assignment to a port and IP address ('host' doesn't matter), set it as current, set the screen to 21x16, and then you should be able to use the stock control and pallet packer programs as if you were on a scanner.

Most problems with non-connection are cause by
- firewalls (turn off to test)
- IPV6 being enabled on Windows 7
- incorrect IP configuration of the Wireless network
- Wavelink not being installed, registered properly, or the monitors not being started as mentioned above

Installing MSSQL 2008 Express

For an overview of the issues to do with installing and updating MSSQL in general, see Installing MSSQL Express Versions. This post covers downloading and installing MSSSL 2008 Express.

MSSQL 2008 is currently at R2 Service Pack 2 (SP2).
Just to keep things nice and confusing, R2 is a new release of SQL Server, not quite a new version, but a major upgrade (MSSQL 2005 is version 9.0, MSSQL 2008 is version 10.0, MSSQL 2008 R2 is version 10.5), and the service packs start over for it as a release.

The release history is:

• MSSQL 2008
• MSSQL 2008 SP1
• MSSQL 2008 SP2
• MSSQL 2008 R2
• MSSQL 2008 R2 SP1
• MSSQL 2008 R2 SP2

Use the R2 SP2 files here.

Note that if you are installing extra features like ReportBuilder (there are a whole list), you'll need to go the R2 SP2 Feature Pack downloads page here rather than download from the product specific R2 pages (again, there is no indication on these product pages that they are outdated).

Use the x64 files if you have a 64 bit Operating System, or the x86 files otherwise.
The top two 'SQLEXPR' files are the bare bones server. The last two, starting in 'SQLEXPRWT', are 'with tools' meaning the server plus Management Studio and a few other development tools.  The two starting with 'SQLEXPRADV' are the whole kit and kaboodle - the server, management studio, business intelligence and reporting services tools.
The SQLEXPR32 file is the 32-bit bare bones server only.

We will follow the installation of the client tools and server version.

(1) Prerequisites - required before commencing installation
The installer will provide you with a download link to the appropriate update.
If you won't be on the internet, then you need to download the files and take them with you.

Installer 4.5 here
  There are Windows 6 (Vista), Windows XP and Server 2003 Versions, x86/x64

PowerShell 2.0 here
This download is not clear - the links are about halfway down the page: 'Windows Management Framework Core (WinRM 2.0 and Windows PowerShell 2.0)'
There are Server 2003, Server 2008, Vista , XP, x86/x64 versions

(2) Task Selection
Choose 'New Installation ...' unless you want to upgrade an existing server

(3) Setup files are installed, the welcome screen is shown, acceptance of Licensing Terms

(4) Feature Selection
Unless you are doing development or specifically using replication, you can leave out these options as shown.

For the core server only install, the page looks like this:

(5) Instance Configuration
I usually install as a named instance.
Usually when I upgrade major versions (2005, 2008, 2008 R2) I install the new server as a separate instance, to allow concurrent running of the systems for a time - hence I name the server to identify the version as shown here.
The old server instance can be removed when no longer needed.
However, you may want to keep the server name consistent and generic (eg. 'SQLExpress') so you don't need to reconfigure the server name on any client machines as it is upgraded through various versions.

(6) Server Configuration
The selection of local service or network service is not crucial.

(7) Windows Authentication Mode is regarded as more secure, but it requires a domain controller and proper network administration.  For legacy Applications which use SQL Authentication (username and password) or for small sites which don't have a dedicated server, Mixed Mode is appropriate.  The password being entered is automatically for the 'sa' (System Administrator) SQL user - you don't have to input the 'sa' username anywhere.

The current windows user should automatically be added to the Administrative users list.

(8) Installation Confirmation and Completion

SQL Server Express 2008 is now installed, but by default it can't be accessed from anywhere but the local computer. To enable remote access:

(1) Go to Start > Program Files > Microsoft SQL Server 2008 R2 > Configuration Tools > SQL Server Configuration Manager

(2) Expand 'SQL Server Network Configuration' and click on 'Protocols for ...' your server instance.  Check 'Shared Memory', 'Named Pipes' and 'TCP/IP'.

(3) You can go to 'SQL Server Services' and restart the SQL Server service, but I'd recommend rebooting the machine at this stage instead.

(4) Any other problems connecting from remote machines can be attributed to firewalls.  Check for firewalls on both the client and server (the Server is critical), and if need be the port to open is port 1433.
Note that the slammer worm uses this port, so what we want to do is enable access to the server machine via the port from the local network only; NOT from the internet.
Firewalls can cause some very strange intermittent behaviour - not just outright blocking - so disabling them completely is the first step in troubleshooting, then they can be progressively re-enabled with appropriate settings, checking for correct operation at each step.

Installing MSSQL 2005 Express

For an overview of the issues to do with installing and updating MSSQL in general, see Installing MSSQL Express Versions. This post covers downloading and installing MSSSL 2005 Express.

The MSSQL 2005 release history is:

• MSSQL 2005
• MSSQL 2005 SP1
• MSSQL 2005 SP2
• MSSQL 2005 SP3
• MSSQL 2005 SP4

It is currently at Service Pack 4 (SP4), downloadable here.

The top file is the core server only, the second file is server + client tools + BI/reporting, and the third is server + client tools.
MSSQL 2005 does not have a native 64 bit installation; but it can be run on 64 bit (it uses the Windows-on-Windows-64 emulation subsystem). The fourth file can be used for native 32-bit-only installs (it leaves out 64 bit compatibility and won't run on 64 bit systems at all).

We will follow the installation of the client tools and server version.

(1) Installing Setup files and welcome screen

 (2) System Configuration Check and Registration Information

(3) Selection of Features

Unless you are doing development or specifically using replication, you can leave out these options as shown.

(4) Windows Authentication Mode is regarded as more secure, but it requires a domain controller and proper network administration.  For legacy Applications which use SQL Authentication (username and password) or for small sites which don't have a dedicated server, Mixed Mode is appropriate.  The password being entered is automatically for the 'sa' (System Administrator) SQL user - you don't have to input the 'sa' username anywhere.

(5) Error Reporting - tick these if you feel inclined.

(6) Installation and Finished Screens

SQL Server Express 2005 is now installed, but by default it can't be accessed from anywhere but the local computer. We need to use the Surface Area Configuration Tool.

(1) Start the tool

(2) Choose 'Surface Area Configuration for Services and Connections'

(3) Make the shown changes and click 'Apply'

(4) You will then need to go to 'Database Engine' > 'Service', and click Restart, then go to 'SQL Server Browser' > 'Service' and click Restart.
This should be all that is needed, but experience shows that it is sometimes necessary to restart the machine, so to save time it's probably a good idea to do that rather than bother restarting the services as just mentioned.

(5) Any other problems connecting from remote machines can be attributed to firewalls.  Check for firewalls on both the client and server (the Server is critical), and if need be the port to open is port 1433.
Note that the slammer worm uses this port, so what we want to do is enable access to the server machine via the port from the local network only; NOT from the internet.

Installing MSSQL Express Versions

You would expect installing the Express version of MS SQL Server to be a straightforward task, but in that you'd be wrong.  Firstly, these are the different released versions of SQL Server Express (as of Oct 2011):

• MSSQL 2005
• MSSQL 2005 SP1
• MSSQL 2005 SP2
• MSSQL 2005 SP3
• MSSQL 2005 SP4

• MSSQL 2008
• MSSQL 2008 SP1
• MSSQL 2008 SP2
• MSSQL 2008 R2
• MSSQL 2008 R2 SP1
• MSSQL 2008 R2 SP2

These are all still available for download, with no warnings on the download page that versions are obsolete.

On top of this, there are three or four different bundles you can download, depending on the features you want - in both 32 (x86) and 64 bit (x64) versions.
The names of features change regularly between versions of MSSQL and sometime even between service packs.

What are your chances of getting the correct latest download ? Not good !

This post will provide links to the combinations you want and guide you through the installation process.

The Ins and Outs of MSSQL Updates

MSSQL service packs have the following behaviour

• updating from an older to newer service pack level essentially reinstalls MSSQL from scratch, so you really don't want to install an old version and update it. However, the advantage of this way of doing things is that you don't have to apply the service packs in order - you can skip straight to the latest
• as soon as you attach your database (mdf) files to the new server, they will be updated to the new server's version format and from that point on the files will not work with an earlier service pack or version of SQL
• for this reason you MUST get the correct server release (2000, 2005, 2008) AND SP level (0-4).  In addition, ALWAYS make a copy of important databases (backups or copies of properly detached mdf files) before installing a Service Pack.

MSSQL Features

There are essentially three features of MSSQL.

• the core server
• client/developer tools (MSSQL Management Studio or MSSMS and a command line query tool) -  these are used during development and/or troubleshooting
• Business Intelligentce (BI)/Anaysis Services and Reporting Services

There will be downloads for (a) core only (b) core + developer tools, (c) core + dev tools + BI/reporting

Checklist

So in summary, you need to decide these points before proceeding:

1. which version of SQL Server Express do you want (2005 or 2008) ?
2. are you installing on a 64 bit or 32 bit Operating System ? (note: this is not necessarily the same as the hardware, as you can run a 32 bit OS on a 64 bit processor)
3. what features do you require from the above list ?

The proceed to the post specific to your version